Re: Re[2]: Solr CPU usage

[Deepak Goel]

1. Are you using two VM's on the same machine?

2. Why are the CPU usage graphs different (during the same time interval
15.40-16.00)? The master and slave are on the same computer, right?

3. The cpu utilised in the method graph is very less as compared to the
time interval shown in the cpu usage graph. Are there any other processes
running on the computer?


I have not used visualvm so its a bit confusing for me (I will have to
download and try it out)



Deepak
"The greatness of a nation can be judged by the way its animals are
treated. Please stop cruelty to Animals, become a Vegan"

+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Wed, May 16, 2018 at 8:28 PM, Александр Шестак <
shestakalexa...@mail.ru.invalid> wrote:

>
> Master/slave are working on single computer with Intel Core i5 3.2GHz (it
> is 4 cores).
>
> It is standard visualVm ui with CPU usage. I think that it show overall
> CPU usage (for all cores).
> >Среда, 16 мая 2018, 17:42 +03:00 от Deepak Goel :
> >
> >How many CPU's do you have in master/slave?
> >
> >The graphs shown by you are for 'a single CPU' or 'All the CPU'?
> >
> >
> >
> >Deepak
> >" The greatness of a nation can be judged by the way its animals are
> treated.  Please stop cruelty to Animals, become a Vegan"
> >
> >+91 73500 12833
> >deic...@gmail.com
> >
> >Facebook:  https://www.facebook.com/deicool
> >LinkedIn:  www.linkedin.com/in/deicool
> >
> >"Plant a Tree, Go Green"
> >
> >Make In India :  http://www.makeinindia.com/home
> >
> >On Wed, May 16, 2018 at 6:41 PM, Александр Шестак  <
> shestakalexa...@mail.ru.invalid > wrote:
> >>Hi, I have a question about unpredictable CPU usage by solr.
> >>We have recently migrated our application from Solr 4.6.1 to Solr 7.1.0.
> We use master/slave approach. And now we have noticed that CPU usage of
> master/slave in passive state (no request to Solr are performed) is none
> zero. When we use Solr 4.6.1 and use tomcat for Solr deploying then CPU
> usage is almost 0. With Solr 7.1.0 cpu usage is varying from 0% to 40% (it
> jumps every time).
> >>
> >>Is it normal behavior for Solr and jetty?
> >>I have tried to analyze this situation by Java VisualVM.
> >>Solr Master CPU usage looks in next way
> >>
> >>Master spend most of time in some jetty method
> >>
> >>Solr Slave CPU usage looks in next way
> >>
> >>
> >>
> >>
> >>
> >>All this screenshots were made when there is no any activity with solr
> performed (solr just was started and no requests to it were performed)
> >
>
>
>
>

Re[2]: Solr CPU usage

[Александр Шестак]

Master/slave are working on single computer with Intel Core i5 3.2GHz (it is 4 
cores).

It is standard visualVm ui with CPU usage. I think that it show overall CPU 
usage (for all cores). 
>Среда, 16 мая 2018, 17:42 +03:00 от Deepak Goel :
>
>How many CPU's do you have in master/slave?
>
>The graphs shown by you are for 'a single CPU' or 'All the CPU'?
>
>
>
>Deepak
>" The greatness of a nation can be judged by the way its animals are treated.  
>Please stop cruelty to Animals, become a Vegan"
>
>+91 73500 12833
>deic...@gmail.com
>
>Facebook:  https://www.facebook.com/deicool
>LinkedIn:  www.linkedin.com/in/deicool
>
>"Plant a Tree, Go Green"
>
>Make In India :  http://www.makeinindia.com/home
>
>On Wed, May 16, 2018 at 6:41 PM, Александр Шестак  < 
>shestakalexa...@mail.ru.invalid > wrote:
>>Hi, I have a question about unpredictable CPU usage by solr.
>>We have recently migrated our application from Solr 4.6.1 to Solr 7.1.0. We 
>>use master/slave approach. And now we have noticed that CPU usage of 
>>master/slave in passive state (no request to Solr are performed) is none 
>>zero. When we use Solr 4.6.1 and use tomcat for Solr deploying then CPU usage 
>>is almost 0. With Solr 7.1.0 cpu usage is varying from 0% to 40% (it jumps 
>>every time).
>>
>>Is it normal behavior for Solr and jetty?
>>I have tried to analyze this situation by Java VisualVM.
>>Solr Master CPU usage looks in next way
>>
>>Master spend most of time in some jetty method
>>
>>Solr Slave CPU usage looks in next way
>>
>>
>>
>>
>>
>>All this screenshots were made when there is no any activity with solr 
>>performed (solr just was started and no requests to it were performed)
>

Re: Solr CPU usage

[Deepak Goel]

How many CPU's do you have in master/slave?

The graphs shown by you are for 'a single CPU' or 'All the CPU'?



Deepak
"The greatness of a nation can be judged by the way its animals are
treated. Please stop cruelty to Animals, become a Vegan"

+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Wed, May 16, 2018 at 6:41 PM, Александр Шестак <
shestakalexa...@mail.ru.invalid> wrote:

> Hi, I have a question about unpredictable CPU usage by solr.
>
> We have recently migrated our application from Solr 4.6.1 to Solr 7.1.0.
> We use master/slave approach. And now we have noticed that CPU usage of
> master/slave in passive state (no request to Solr are performed) is none
> zero. When we use Solr 4.6.1 and use tomcat for Solr deploying then CPU
> usage is almost 0. With Solr 7.1.0 cpu usage is varying from 0% to 40% (it
> jumps every time).
>
>
> Is it normal behavior for Solr and jetty?
>
> I have tried to analyze this situation by Java VisualVM.
>
> Solr Master CPU usage looks in next way
>
>
> Master spend most of time in some jetty method
>
>
> Solr Slave CPU usage looks in next way
>
>
>
>
>
>
> All this screenshots were made when there is no any activity with solr
> performed (solr just was started and no requests to it were performed)
>
>

Re: Solr CPU usage

[Shawn Heisey]

On 5/16/2018 7:11 AM, Александр Шестак wrote:


Hi, I have a question about unpredictable CPU usage by solr.

We have recently migrated our application from Solr 4.6.1 to Solr 
7.1.0. We use master/slave approach. And now we have noticed that CPU 
usage of master/slave in passive state (no request to Solr are 
performed) is none zero. When we use Solr 4.6.1 and use tomcat for 
Solr deploying then CPU usage is almost 0. With Solr 7.1.0 cpu usage 
is varying from 0% to 40% (it jumps every time).




The information you've obtained about where the CPU time is spent shows 
that your master server is spending its CPU time in Jetty code, not Solr 
code.  I do not know why.  Because it's code not written by this 
project, there may not be much we can do about it, other than encourage 
you to upgrade Solr so that you have a newer version of Jetty.  Version 
7.1.0 has Jetty 9.3.20, but 7.3.0 and 7.3.1 have Jetty 9.4.8.


The slave appears to be spending time in Lucene's merge code.  Which is 
very odd ... merging should normally happen on the master, usually not 
on slaves.


Thanks,
Shawn

Re: Solr CPU Usage

[hendra_budiawan]

Yes we do complex query with a lot of clauses and facets and data is growing
up bigger every day, i agree with you it might not on the hardware issue
maybe i need to tune up solr/OS/jetty system configration to optimize solr
process. Thank you so much for help.

Best regards,
Hendra



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155528.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Solr CPU Usage

[Jacques du Rand]

Do you index and search from this box ?
How many documents do you have ?


From: Shawn Heisey [s...@elyograg.org]
Sent: Thursday, August 28, 2014 7:48 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr CPU Usage

On 8/27/2014 8:42 PM, hendra_budiawan wrote:
 Yes i'm just worried about load average reported by OS, because last week
 suddenly server can't accessed  so we have to hard reboot. I'm still
 investigating what is the problem, because this server is dedicated to solr
 only, we suspect the problem came from the solr process but i'm still
 looking another possibility what makes this problem arises. Can you give me
 suggestion what supposed i need to check further?

What kind of query volume is your Solr server supporting?  Are you doing
complex queries with a lot of clauses, facets, or something else that's
CPU intensive?  Is your update volume high?

The numbers that you've shown, assuming that the htop info is accurate
and you really do have 16 or 32 CPU cores, do not look like any major
problem.  Solr is working hard, but there's a lot more CPU capacity
left.  The top output shows that iowait percentage is not a problem, so
it's not stuck in disk I/O.  Memory usage indicates that OS disk caching
is working well.

It looks like you were running jetty, but that the jetty might not be
the one included in the Solr example.  If it's not the one included in
the example, then its configuration is not well-tuned for Solr.  If you
have a high request volume, you may need to increase the maxThreads
parameter in the jetty config.  The only possible thing that I can think
of which might cause a complete inability to access the server via ssh
or other means is that you are hitting the open file limit in the
operating system.  Most linux distros use /etc/security/limits.conf to
configure the open file limit for each user.

Thanks,
Shawn

This email and its contents are subject to an email legal notice that can be 
viewed at http://www.naspers.com/disclaimer.php Should you be unable to access 
the link provided, please email us for a copy at c...@optinet.net
Hierdie e-pos en sy inhoud is onderhewig aan 'n regskennisgewing oor 
elektroniese pos wat gelees kan word by 
http://www.naspers.com/afrikaans/voorbehoud.php 'n Afskrif kan aangevra word by 
c...@optinet.net

RE: Solr CPU Usage

[hendra_budiawan]

HI Jacques,

Yes we index and search from this box, we have 6 core with almost 4000K
document each core getting and bigger each day. 

Regards,
Hendra Budiawan



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p412.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Solr CPU Usage

[Jacques du Rand]

HI Hendra
That doesn't seem overly huge...

I agree with the other person saying, from the top/htop graph it doesnt look 
too bad.
I will maybe try to split the searching/indexing as well try to schedule the 
delta index for the cores at different times maybe


PS.We had a nice little bump in efficiency by going with Tomcat 7 and java8.

Jacques


From: hendra_budiawan [hendra.budiawan...@gmail.com]
Sent: Thursday, August 28, 2014 9:46 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr CPU Usage

HI Jacques,

Yes we index and search from this box, we have 6 core with almost 4000K
document each core getting and bigger each day.

Regards,
Hendra Budiawan



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p412.html
Sent from the Solr - User mailing list archive at Nabble.com.
This email and its contents are subject to an email legal notice that can be 
viewed at http://www.naspers.com/disclaimer.php Should you be unable to access 
the link provided, please email us for a copy at c...@optinet.net
Hierdie e-pos en sy inhoud is onderhewig aan 'n regskennisgewing oor 
elektroniese pos wat gelees kan word by 
http://www.naspers.com/afrikaans/voorbehoud.php 'n Afskrif kan aangevra word by 
c...@optinet.net

RE: Solr CPU Usage

[hendra_budiawan]

Hi Jacques,

I will try your advice to schedule index with different times also will try
to start research with tomcat7 and java8.

Thank you so much,
Hendra



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155562.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[Chris Hostetter]

: Yes i'm just worried about load average reported by OS, because last week
: suddenly server can't accessed  so we have to hard reboot. I'm still
: investigating what is the problem, because this server is dedicated to solr

ok - so here is the key bit.

basically, nothing else you've mentioend in this thread indicates any sort 
of problem -- your load (now, the one you've observed) is fine.  the 
questionsi what happened last week?

do you have any metrics/monitoring information from the server when you 
actually had a problem?

do you have any logs (from Solr, or from jetty, or from the OS, or any 
OS/hardware monitoring tools from last week when the problem happened?

define server can't accessed?  do you mean solr wasn't responding to 
queries, or do you mean i couldn't evevne ping the machine, let alone ssh 
to it? ... because there is a big difference.

if you can ssh to a machine, but solr is not responding, then generating 
thread dumps would help see what solr is doing.


-Hoss
http://www.lucidworks.com/

Re: Solr CPU Usage

[rulinma]

I think that is configs not tuned well. 
Can use jmx to monitor what is doing?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155747.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[Greg Harris]

Here is a quick way you can identify which thread is taking up all your CPU.

1) Look at top (or htop) sorted by CPU Usage and with threads toggled on -
hit capital 'H'
2) Get the native process ids of the threads taking up a lot of CPU
3) Convert that number to hex using a converter:
http://www.mathsisfun.com/binary-decimal-hexadecimal-converter.html
4) Use the hex number to identify the problematic threads on the thread
dump via the nid= value. So for example:
nid=0x549 would equate to the native thread id of 1353 on top.

Take a thread dump and identify any problematic threads so you can see the
stack trace.
However, Chris has pointed out that there is as of yet no evidence your
outage is related to CPU overload.

Greg

On Thu, Aug 28, 2014 at 6:45 PM, rulinma ruli...@gmail.com wrote:

 I think that is configs not tuned well.
 Can use jmx to monitor what is doing?



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155747.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Solr CPU Usage

[hendra_budiawan]

Hi all,

I'm having high cpu usage on my server, detailed on picture below
http://lucene.472066.n3.nabble.com/file/n4155370/htop-server.png 

Using default config for solrconfig.xml  schema.xml, can anyone help me to
identified why the cpu so high on solr process?

Thank's in advanced.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[Shawn Heisey]

On 8/27/2014 4:16 AM, hendra_budiawan wrote:
 I'm having high cpu usage on my server, detailed on picture below
 http://lucene.472066.n3.nabble.com/file/n4155370/htop-server.png 
 
 Using default config for solrconfig.xml  schema.xml, can anyone help me to
 identified why the cpu so high on solr process?

A standard top screenshot would be a lot more useful than htop -- it
includes information about memory sizes and utilization.

The most common reason for performance issues is not enough RAM, either
heap or OS disk cache, maybe both.  Let's start with a standard top
screenshot, then additional questions may be required from there.  Some
light reading in the meantime:

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn

Re: Solr CPU Usage

[Jack Krupansky]

Is the high usage just suddenly happening after a long period of up-time 
without it, or is this on a server restart? The latter can happen if you 
have a large commit log to replay because you haven't done hard commits.


-- Jack Krupansky

-Original Message- 
From: Shawn Heisey

Sent: Wednesday, August 27, 2014 9:51 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr CPU Usage

On 8/27/2014 4:16 AM, hendra_budiawan wrote:

I'm having high cpu usage on my server, detailed on picture below
http://lucene.472066.n3.nabble.com/file/n4155370/htop-server.png

Using default config for solrconfig.xml  schema.xml, can anyone help me 
to

identified why the cpu so high on solr process?


A standard top screenshot would be a lot more useful than htop -- it
includes information about memory sizes and utilization.

The most common reason for performance issues is not enough RAM, either
heap or OS disk cache, maybe both.  Let's start with a standard top
screenshot, then additional questions may be required from there.  Some
light reading in the meantime:

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn 

Re: Solr CPU Usage

[hendra_budiawan]

HI Shawn, thank's for reply here is standard top

http://lucene.472066.n3.nabble.com/file/n4155505/top.png 

My server have enough RAM, so i think it's not RAMproblem

 total   used   free sharedbuffers cached
Mem: 32073  12629  19443  0293   5549
-/+ buffers/cache:   6786  25286
Swap:31999  0  31999

Still investigating right know to find the problem.


Regards,
Hendra Budiawan



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155505.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[hendra_budiawan]

Hi Jack,

High usage is happening when we restart the server a week ago, we still
investigating to find the problem maybe there is something i'm missed to
check.

Regards,
Hendra Budiawan



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155507.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[Shawn Heisey]

On 8/27/2014 7:53 PM, hendra_budiawan wrote:
 HI Shawn, thank's for reply here is standard top
 
 http://lucene.472066.n3.nabble.com/file/n4155505/top.png 
 
 My server have enough RAM, so i think it's not RAMproblem
 
  total   used   free sharedbuffers cached
 Mem: 32073  12629  19443  0293   5549
 -/+ buffers/cache:   6786  25286
 Swap:31999  0  31999

For a machine with 32 CPU cores (basing this on what I saw on the htop
screen), a system load of 5-6 (and a CPU usage of 600%) doesn't really
strike me as a problem.  If you actually have 16 hyper-threaded CPU
cores, it's STILL not a problem.  On a system with one or two CPU cores,
it would be a VERY high load.  If you don't actually have that many CPU
cores, please let us know ... your hardware may be lying to Linux.

http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

I also don't see any problems in your memory usage numbers.

Are you just worried about the load average numbers reported by your OS,
or are you actually seeing problems with your Solr server?

Thanks,
Shawn

Re: Solr CPU Usage

[hendra_budiawan]

Hi Shawn, 

Yes i'm just worried about load average reported by OS, because last week
suddenly server can't accessed  so we have to hard reboot. I'm still
investigating what is the problem, because this server is dedicated to solr
only, we suspect the problem came from the solr process but i'm still
looking another possibility what makes this problem arises. Can you give me
suggestion what supposed i need to check further?

Thank's in advanced
Hendra



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155512.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr CPU Usage

[Shawn Heisey]

On 8/27/2014 8:42 PM, hendra_budiawan wrote:
 Yes i'm just worried about load average reported by OS, because last week
 suddenly server can't accessed  so we have to hard reboot. I'm still
 investigating what is the problem, because this server is dedicated to solr
 only, we suspect the problem came from the solr process but i'm still
 looking another possibility what makes this problem arises. Can you give me
 suggestion what supposed i need to check further?

What kind of query volume is your Solr server supporting?  Are you doing
complex queries with a lot of clauses, facets, or something else that's
CPU intensive?  Is your update volume high?

The numbers that you've shown, assuming that the htop info is accurate
and you really do have 16 or 32 CPU cores, do not look like any major
problem.  Solr is working hard, but there's a lot more CPU capacity
left.  The top output shows that iowait percentage is not a problem, so
it's not stuck in disk I/O.  Memory usage indicates that OS disk caching
is working well.

It looks like you were running jetty, but that the jetty might not be
the one included in the Solr example.  If it's not the one included in
the example, then its configuration is not well-tuned for Solr.  If you
have a high request volume, you may need to increase the maxThreads
parameter in the jetty config.  The only possible thing that I can think
of which might cause a complete inability to access the server via ssh
or other means is that you are hitting the open file limit in the
operating system.  Most linux distros use /etc/security/limits.conf to
configure the open file limit for each user.

Thanks,
Shawn

Re: solr cpu usage

[Tim Vaillancourt]

Yes, you've saved us all lots of time with this article. I'm about to do
the same for the old Jetty or Tomcat? container question ;).

Tim


On 7 October 2013 18:55, Erick Erickson erickerick...@gmail.com wrote:

 Tim:

 Thanks! Mostly I wrote it to have something official looking to hide
 behind when I didn't have a good answer to the hardware sizing question
 :).

 On Mon, Oct 7, 2013 at 2:48 PM, Tim Vaillancourt t...@elementspace.com
 wrote:
  Fantastic article!
 
  Tim
 
 
  On 5 October 2013 18:14, Erick Erickson erickerick...@gmail.com wrote:
 
  From my perspective, your question is almost impossible to
  answer, there are too many variables. See:
 
 
 http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
 
  Best,
  Erick
 
  On Thu, Oct 3, 2013 at 9:38 PM, Otis Gospodnetic
  otis.gospodne...@gmail.com wrote:
   Hi,
  
   More CPU cores means more concurrency.  This is good if you need to
  handle
   high query rates.
  
   Faster cores mean lower query latency, assuming you are not
 bottlenecked
  by
   memory or disk IO or network IO.
  
   So what is ideal for you depends on your concurrency and latency
 needs.
  
   Otis
   Solr  ElasticSearch Support
   http://sematext.com/
   On Oct 1, 2013 9:33 AM, adfel70 adfe...@gmail.com wrote:
  
   hi
   We're building a spec for a machine to purchase.
   We're going to buy 10 machines.
   we aren't sure yet how many proccesses we will run per machine.
   the question is  -should we buy faster cpu with less cores or slower
 cpu
   with more cores?
   in any case we will have 2 cpus in each machine.
   should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?
  
   what will we gain by having many cores?
  
   what kinds of usages would make cpu be the bottleneck?
  
  
  
  
   --
   View this message in context:
   http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
   Sent from the Solr - User mailing list archive at Nabble.com.
  
 

Re: solr cpu usage

[Tim Vaillancourt]

Fantastic article!

Tim


On 5 October 2013 18:14, Erick Erickson erickerick...@gmail.com wrote:

 From my perspective, your question is almost impossible to
 answer, there are too many variables. See:

 http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

 Best,
 Erick

 On Thu, Oct 3, 2013 at 9:38 PM, Otis Gospodnetic
 otis.gospodne...@gmail.com wrote:
  Hi,
 
  More CPU cores means more concurrency.  This is good if you need to
 handle
  high query rates.
 
  Faster cores mean lower query latency, assuming you are not bottlenecked
 by
  memory or disk IO or network IO.
 
  So what is ideal for you depends on your concurrency and latency needs.
 
  Otis
  Solr  ElasticSearch Support
  http://sematext.com/
  On Oct 1, 2013 9:33 AM, adfel70 adfe...@gmail.com wrote:
 
  hi
  We're building a spec for a machine to purchase.
  We're going to buy 10 machines.
  we aren't sure yet how many proccesses we will run per machine.
  the question is  -should we buy faster cpu with less cores or slower cpu
  with more cores?
  in any case we will have 2 cpus in each machine.
  should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?
 
  what will we gain by having many cores?
 
  what kinds of usages would make cpu be the bottleneck?
 
 
 
 
  --
  View this message in context:
  http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
  Sent from the Solr - User mailing list archive at Nabble.com.
 

Re: solr cpu usage

[Erick Erickson]

Tim:

Thanks! Mostly I wrote it to have something official looking to hide
behind when I didn't have a good answer to the hardware sizing question
:).

On Mon, Oct 7, 2013 at 2:48 PM, Tim Vaillancourt t...@elementspace.com wrote:
 Fantastic article!

 Tim


 On 5 October 2013 18:14, Erick Erickson erickerick...@gmail.com wrote:

 From my perspective, your question is almost impossible to
 answer, there are too many variables. See:

 http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

 Best,
 Erick

 On Thu, Oct 3, 2013 at 9:38 PM, Otis Gospodnetic
 otis.gospodne...@gmail.com wrote:
  Hi,
 
  More CPU cores means more concurrency.  This is good if you need to
 handle
  high query rates.
 
  Faster cores mean lower query latency, assuming you are not bottlenecked
 by
  memory or disk IO or network IO.
 
  So what is ideal for you depends on your concurrency and latency needs.
 
  Otis
  Solr  ElasticSearch Support
  http://sematext.com/
  On Oct 1, 2013 9:33 AM, adfel70 adfe...@gmail.com wrote:
 
  hi
  We're building a spec for a machine to purchase.
  We're going to buy 10 machines.
  we aren't sure yet how many proccesses we will run per machine.
  the question is  -should we buy faster cpu with less cores or slower cpu
  with more cores?
  in any case we will have 2 cpus in each machine.
  should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?
 
  what will we gain by having many cores?
 
  what kinds of usages would make cpu be the bottleneck?
 
 
 
 
  --
  View this message in context:
  http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
  Sent from the Solr - User mailing list archive at Nabble.com.
 

Re: solr cpu usage

[Erick Erickson]

From my perspective, your question is almost impossible to
answer, there are too many variables. See:
http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Best,
Erick

On Thu, Oct 3, 2013 at 9:38 PM, Otis Gospodnetic
otis.gospodne...@gmail.com wrote:
 Hi,

 More CPU cores means more concurrency.  This is good if you need to handle
 high query rates.

 Faster cores mean lower query latency, assuming you are not bottlenecked by
 memory or disk IO or network IO.

 So what is ideal for you depends on your concurrency and latency needs.

 Otis
 Solr  ElasticSearch Support
 http://sematext.com/
 On Oct 1, 2013 9:33 AM, adfel70 adfe...@gmail.com wrote:

 hi
 We're building a spec for a machine to purchase.
 We're going to buy 10 machines.
 we aren't sure yet how many proccesses we will run per machine.
 the question is  -should we buy faster cpu with less cores or slower cpu
 with more cores?
 in any case we will have 2 cpus in each machine.
 should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?

 what will we gain by having many cores?

 what kinds of usages would make cpu be the bottleneck?




 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr cpu usage

[Otis Gospodnetic]

Hi,

More CPU cores means more concurrency.  This is good if you need to handle
high query rates.

Faster cores mean lower query latency, assuming you are not bottlenecked by
memory or disk IO or network IO.

So what is ideal for you depends on your concurrency and latency needs.

Otis
Solr  ElasticSearch Support
http://sematext.com/
On Oct 1, 2013 9:33 AM, adfel70 adfe...@gmail.com wrote:

 hi
 We're building a spec for a machine to purchase.
 We're going to buy 10 machines.
 we aren't sure yet how many proccesses we will run per machine.
 the question is  -should we buy faster cpu with less cores or slower cpu
 with more cores?
 in any case we will have 2 cpus in each machine.
 should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?

 what will we gain by having many cores?

 what kinds of usages would make cpu be the bottleneck?




 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
 Sent from the Solr - User mailing list archive at Nabble.com.

solr cpu usage

[adfel70]

hi
We're building a spec for a machine to purchase.
We're going to buy 10 machines.
we aren't sure yet how many proccesses we will run per machine.
the question is  -should we buy faster cpu with less cores or slower cpu
with more cores?
in any case we will have 2 cpus in each machine.
should we buy 2.6Ghz cpu with 8 cores or 3.5Ghz cpu with 4 cores?

what will we gain by having many cores?

what kinds of usages would make cpu be the bottleneck?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-cpu-usage-tp4092938.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr CPU usage

[Girish Redekar]

Hi

I'm testing my Solr instance with multiple simultaneous requests. Here's my
test.

For an index of ~200K docs, I query Solr with 10 simultaneous threads. Can
someone help me explain/improve the following observations:

1) Solr doesn't seem to use all the available CPU to improve response times
(query times are good, but the time required to return documents aren't so
good). My CPU seems to be running at ~30%
2) As expected, time for response increases as the num of requested results
increase. What's surprising (and perplexing) is that Solr seems to use
*more* of the CPU when I'm requesting *fewer* docs. Consequently, its
performance in returning a larger result set is very bad
3) To counter 1, is there a way to make two Solr instances search on the
same index (so that concurrent requests are served faster)

Any help in this regard would be very useful.

Thanks !
Girish Redekar
http://girishredekar.net

Re: Solr CPU usage

[Girish Redekar]

Yonik,
Am running both my server and client on ubuntu machines. The client is on a
different box. The server CPU and RAM are both well below 50%.

Girish Redekar
http://girishredekar.net


On Fri, Nov 27, 2009 at 10:07 PM, Yonik Seeley
yo...@lucidimagination.comwrote:

 On Fri, Nov 27, 2009 at 9:30 AM, Girish Redekar
 girish.rede...@aplopio.com wrote:
  Hi
 
  I'm testing my Solr instance with multiple simultaneous requests. Here's
 my
  test.
 
  For an index of ~200K docs, I query Solr with 10 simultaneous threads.
 Can
  someone help me explain/improve the following observations:
 
  1) Solr doesn't seem to use all the available CPU to improve response
 times
  (query times are good, but the time required to return documents aren't
 so
  good). My CPU seems to be running at ~30%
  2) As expected, time for response increases as the num of requested
 results
  increase. What's surprising (and perplexing) is that Solr seems to use
  *more* of the CPU when I'm requesting *fewer* docs. Consequently, its
  performance in returning a larger result set is very bad

 This may point to another bottleneck - if the OS is low on free RAM,
 it could be disk IO.
 If this is on Windows, you could have contention reading the index files.
 Otherwise, you may have a bottleneck in network IO.  Is the client you
 are testing with on the same box?
 Is this Solr 1.4?

 -Yonik
 http://www.lucidimagination.com

Re: Solr CPU usage

[Yonik Seeley]

On Fri, Nov 27, 2009 at 12:09 PM, Girish Redekar
girish.rede...@aplopio.com wrote:
 Am running both my server and client on ubuntu machines. The client is on a
 different box. The server CPU and RAM are both well below 50%.

OK, then the obvious thing to try would be to move the client to the
server machine and see if you can max out the CPUs.  Sould be doable
on Solr 1.4

-Yonik
http://www.lucidimagination.com