Re: Solr4 To Solr6 CPU load issues
On Sun, Feb 11, 2018 at 8:47 AM, ~$alpha`wrote: > I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but > overall the CPU load on solr6.6 is double the load on solr4.0 and hence I am > not able to roll solr6.6 to 100% of my traffic. > > *Some Key Stats In Performance of Sol6 Vs Solr4* > Document cache usage increased from .98 from .14 > Query Result cache usage increased from .10 from .24 > Filter cache same as .94 > Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field > multivalued concept was changed from solr4 to solr6) It could be faceting... "uif" was the default faceting method in Solr 4 (which used fieldValueCache), which was good for facet performance but bad for heap usage and index turnaround time (NRT). It was removed, but then later re-added in the JSON Facet API. Access to that was hooked into the older facet API via facet.method=uif https://issues.apache.org/jira/browse/SOLR-8466 -Yonik
Re: Solr4 To Solr6 CPU load issues
cpu utlization is already on higher end, so VM wont seems a solution -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
One more idea could be is to have multiple vm's (8 cpu each) on your server and load balance them. That would help Solr6 scale nicely On 12 Feb 2018 23:05, "Deepak Goel"wrote: > If the community cannot help, the only way i can think is either to > profile Solr (java) under a load test to find the problem. You could also > use an APM. > > On 12 Feb 2018 23:00, "~$alpha`" wrote: > >> Yes, but how to move ahead now. >> Its strange solr4 is better behaving than solr6 >> >> >> >> -- >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >> >
Re: Solr4 To Solr6 CPU load issues
top command in linux -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
25 of 32 CPUs loaded is 78% CPU, but I’ve never seen CPU use reported that way. How are you measuring CPU usage? What tool? wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Feb 12, 2018, at 7:08 AM, ~$alpha`wrote: > > I cant test on more as performance is already degraded. > Its a 32core system and load 25 means 2500% cpu > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
If the community cannot help, the only way i can think is either to profile Solr (java) under a load test to find the problem. You could also use an APM. On 12 Feb 2018 23:00, "~$alpha`"wrote: > Yes, but how to move ahead now. > Its strange solr4 is better behaving than solr6 > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Solr4 To Solr6 CPU load issues
Yes, but how to move ahead now. Its strange solr4 is better behaving than solr6 -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
This would then mean that solr6 is reaching some kind of saturation (number of threads, etc) at about loads of 60 Hits which then drives the performance of it to be very bad ! Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Mon, Feb 12, 2018 at 8:38 PM, ~$alpha`wrote: > I cant test on more as performance is already degraded. > Its a 32core system and load 25 means 2500% cpu > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Solr4 To Solr6 CPU load issues
100% CPU will cause congestion and very slow response. In production, we do not drive Solr over 75% CPU. He is reporting load average, which is a bit harder to interpret. When the load average reaches the number of CPUs, that is probably the beginning of congestion. But I’m less sure about that than about the 75% CPU level. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Feb 12, 2018, at 6:17 AM, Deepak Goelwrote: > > Please test for higher number of hits till cpu load reaches 100% > > On 12 Feb 2018 19:44, "~$alpha`" wrote: > >> Hits 41 : >> Avg response time470ms vs 380ms >> CPU Load reaches6 vs 10 >> >> Hits 82: >> Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) >> CPU Load reaches11 vs 25 >> >> >> >> -- >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >>
Re: Solr4 To Solr6 CPU load issues
I cant test on more as performance is already degraded. Its a 32core system and load 25 means 2500% cpu -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Please test for higher number of hits till cpu load reaches 100% On 12 Feb 2018 19:44, "~$alpha`"wrote: > Hits 41 : > Avg response time470ms vs 380ms > CPU Load reaches6 vs 10 > > Hits 82: > Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) > CPU Load reaches11 vs 25 > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Solr4 To Solr6 CPU load issues
Hits 41 : Avg response time470ms vs 380ms CPU Load reaches6 vs 10 Hits 82: Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) CPU Load reaches11 vs 25 -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
I would suggest to keep the load same both for solr4 and solr6. And then test. Also please post exact concurrent hits On 12 Feb 2018 12:48, "~$alpha`"wrote: When both solr4 and solr6 have concurrent hits: 1. 30 to 40 : Avg response time 470ms vs 380ms Load 6 vs 10 1. 80 to 90 : Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) Load 11 vs 25 -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Hi Birender, Do you monitor you heap? Is it possible that you are running close to max heap size and that GC is what is taking CPU? Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 11 Feb 2018, at 19:38, Birender Rawatwrote: > > I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but > overall the CPU load on solr6.6 is double the load on solr4.0 and hence I > am not able to roll solr6.6 to 100% of my traffic. > > *Some Key Stats In Performance of Sol6 Vs Solr4* > Document cache usage increased from .98 from .14 > Query Result cache usage increased from .10 from .24 > Filter cache same as .94 > Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field > multivalued concept was changed from solr4 to solr6) > > *Please Help Note: I have given document cache 3 times memory for > doc.cache.*
Re: Solr4 To Solr6 CPU load issues
When both solr4 and solr6 have concurrent hits: 1. 30 to 40 : Avg response time 470ms vs 380ms Load 6 vs 10 1. 80 to 90 : Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) Load 11 vs 25 -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Also can you please post the throughputs for both of your tests On 12 Feb 2018 00:35, "Deepak Goel"wrote: > Yup. Improvement of response time would hurt the cpu usage. The other > thing is more memory usage (cache) which gets included into the cpu usage. > > > > Deepak > "Please stop cruelty to Animals, help by becoming a Vegan" > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha` wrote: > >> Other resource is not using the cpu. >> Its true that response is better to 300ms from 350ms but cpu usage almost >> doubled? >> >> >> >> -- >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >> > >
Re: Solr4 To Solr6 CPU load issues
Yup. Improvement of response time would hurt the cpu usage. The other thing is more memory usage (cache) which gets included into the cpu usage. Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha`wrote: > Other resource is not using the cpu. > Its true that response is better to 300ms from 350ms but cpu usage almost > doubled? > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Solr4 To Solr6 CPU load issues
Other resource is not using the cpu. Its true that response is better to 300ms from 350ms but cpu usage almost doubled? -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Two things trouble me: 1. It is a shared resource, so results are unreliable 2. Since cache results have increased, memory access will increase and it will result in an increase in cpu usage. However response times will also improve To support more load you will have to increase server capacity or add servers. On 11 Feb 2018 23:28, "~$alpha`"wrote: Config : 64GB RAM 32 CORE CPU but i have given 20Gb to solr JVM.. Also its a shared resource -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Config : 64GB RAM 32 CORE CPU but i have given 20Gb to solr JVM.. Also its a shared resource -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr4 To Solr6 CPU load issues
Can you please give the configuration of your server? On 11 Feb 2018 19:17, "~$alpha`"wrote: > I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but > overall the CPU load on solr6.6 is double the load on solr4.0 and hence I > am > not able to roll solr6.6 to 100% of my traffic. > > *Some Key Stats In Performance of Sol6 Vs Solr4* > Document cache usage increased from .98 from .14 > Query Result cache usage increased from .10 from .24 > Filter cache same as .94 > Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field > multivalued concept was changed from solr4 to solr6) > > *Please Help Note: I have given document cache 3 times memory for > doc.cache.* > > > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >