Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Yonik Seeley
On Sun, Feb 11, 2018 at 8:47 AM, ~$alpha`  wrote:
> I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
> overall the CPU load on solr6.6 is double the load on solr4.0 and hence I am
> not able to roll solr6.6 to 100% of my traffic.
>
> *Some Key Stats In Performance of Sol6 Vs Solr4*
> Document cache usage increased from .98 from .14
> Query Result cache usage increased from .10 from .24
> Filter cache same as .94
> Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
> multivalued concept was changed from solr4 to solr6)

It could be faceting... "uif" was the default faceting method in Solr
4 (which used fieldValueCache), which was good for facet performance
but bad for heap usage and index turnaround time (NRT).
It was removed, but then later re-added in the JSON Facet API.  Access
to that was hooked into the older facet API via facet.method=uif
https://issues.apache.org/jira/browse/SOLR-8466

-Yonik


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread ~$alpha`
cpu utlization is already on higher end, so VM wont seems a solution



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Deepak Goel
One more idea could be is to have multiple vm's (8 cpu each) on your server
and load balance them. That would help Solr6 scale nicely

On 12 Feb 2018 23:05, "Deepak Goel"  wrote:

> If the community cannot help, the only way i can think is either to
> profile Solr (java) under a load test to find the problem. You could also
> use an APM.
>
> On 12 Feb 2018 23:00, "~$alpha`"  wrote:
>
>> Yes, but how to move ahead  now.
>> Its strange solr4 is better behaving than solr6
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread ~$alpha`
top command in linux 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Walter Underwood
25 of 32 CPUs loaded is 78% CPU, but I’ve never seen CPU use reported that way.

How are you measuring CPU usage? What tool?

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 12, 2018, at 7:08 AM, ~$alpha`  wrote:
> 
> I cant test on more as performance is already degraded.
> Its a 32core system and load 25 means 2500% cpu
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html



Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Deepak Goel
If the community cannot help, the only way i can think is either to profile
Solr (java) under a load test to find the problem. You could also use an
APM.

On 12 Feb 2018 23:00, "~$alpha`"  wrote:

> Yes, but how to move ahead  now.
> Its strange solr4 is better behaving than solr6
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread ~$alpha`
Yes, but how to move ahead  now.
Its strange solr4 is better behaving than solr6



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Deepak Goel
This would then mean that solr6 is reaching some kind of saturation (number
of threads, etc) at about loads of 60 Hits which then drives the
performance of it to be very bad !



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Mon, Feb 12, 2018 at 8:38 PM, ~$alpha`  wrote:

> I cant test on more as performance is already degraded.
> Its a 32core system and load 25 means 2500% cpu
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Walter Underwood
100% CPU will cause congestion and very slow response.

In production, we do not drive Solr over 75% CPU.

He is reporting load average, which is a bit harder to interpret. When the load 
average reaches the number of CPUs, that is probably the beginning of 
congestion. But I’m less sure about that than about the 75% CPU level.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 12, 2018, at 6:17 AM, Deepak Goel  wrote:
> 
> Please test for higher number of hits till cpu load reaches 100%
> 
> On 12 Feb 2018 19:44, "~$alpha`"  wrote:
> 
>> Hits 41 :
>> Avg response time470ms vs 380ms
>> CPU Load  reaches6 vs 10
>> 
>> Hits 82:
>> Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
>> CPU Load  reaches11 vs 25
>> 
>> 
>> 
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>> 



Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread ~$alpha`
I cant test on more as performance is already degraded.
Its a 32core system and load 25 means 2500% cpu



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Deepak Goel
Please test for higher number of hits till cpu load reaches 100%

On 12 Feb 2018 19:44, "~$alpha`"  wrote:

> Hits 41 :
> Avg response time470ms vs 380ms
> CPU Load  reaches6 vs 10
>
> Hits 82:
> Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
> CPU Load  reaches11 vs 25
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread ~$alpha`
Hits 41 : 
Avg response time470ms vs 380ms 
CPU Load  reaches6 vs 10 

Hits 82: 
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) 
CPU Load  reaches11 vs 25



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Deepak Goel
I would suggest to keep the load same both for solr4 and solr6. And then
test. Also please post exact concurrent hits

On 12 Feb 2018 12:48, "~$alpha`"  wrote:

When both solr4 and solr6 have concurrent hits:
1. 30 to 40 :
Avg response time 470ms vs 380ms
Load 6 vs 10

1. 80 to 90 :
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
Load 11 vs 25




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-12 Thread Emir Arnautović
Hi Birender,
Do you monitor you heap? Is it possible that you are running close to max heap 
size and that GC is what is taking CPU?

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Feb 2018, at 19:38, Birender Rawat  wrote:
> 
> I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
> overall the CPU load on solr6.6 is double the load on solr4.0 and hence I
> am not able to roll solr6.6 to 100% of my traffic.
> 
> *Some Key Stats In Performance of Sol6 Vs Solr4*
> Document cache usage increased from .98 from .14
> Query Result cache usage increased from .10 from .24
> Filter cache same as .94
> Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
> multivalued concept was changed from solr4 to solr6)
> 
> *Please Help Note: I have given document cache 3 times memory for
> doc.cache.*



Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread ~$alpha`
When both solr4 and solr6 have concurrent hits:
1. 30 to 40 : 
Avg response time 470ms vs 380ms
Load 6 vs 10

1. 80 to 90 : 
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
Load 11 vs 25




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread Deepak Goel
Also can you please post the throughputs for both of your tests

On 12 Feb 2018 00:35, "Deepak Goel"  wrote:

> Yup. Improvement of response time would hurt the cpu usage. The other
> thing is more memory usage (cache) which gets included into the cpu usage.
>
>
>
> Deepak
> "Please stop cruelty to Animals, help by becoming a Vegan"
> +91 73500 12833
> deic...@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha`  wrote:
>
>> Other resource is not using the cpu.
>> Its true that response is better to 300ms from 350ms but cpu usage almost
>> doubled?
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>
>


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread Deepak Goel
Yup. Improvement of response time would hurt the cpu usage. The other thing
is more memory usage (cache) which gets included into the cpu usage.



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha`  wrote:

> Other resource is not using the cpu.
> Its true that response is better to 300ms from 350ms but cpu usage almost
> doubled?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread ~$alpha`
Other resource is not using the cpu.
Its true that response is better to 300ms from 350ms but cpu usage almost
doubled?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread Deepak Goel
Two things trouble me:

1. It is a shared resource, so results are unreliable

2. Since cache results have increased, memory access will increase and it
will result in an increase in cpu usage. However response times will also
improve

To support more load you will have to increase server capacity or add
servers.




On 11 Feb 2018 23:28, "~$alpha`"  wrote:

Config : 64GB RAM 32 CORE CPU
but i have given 20Gb to solr JVM.. Also its a shared resource



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread ~$alpha`
Config : 64GB RAM 32 CORE CPU
but i have given 20Gb to solr JVM.. Also its a shared resource



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html


Re: Solr4 To Solr6 CPU load issues

2018-02-11 Thread Deepak Goel
Can you please give the configuration of your server?

On 11 Feb 2018 19:17, "~$alpha`"  wrote:

> I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
> overall the CPU load on solr6.6 is double the load on solr4.0 and hence I
> am
> not able to roll solr6.6 to 100% of my traffic.
>
> *Some Key Stats In Performance of Sol6 Vs Solr4*
> Document cache usage increased from .98 from .14
> Query Result cache usage increased from .10 from .24
> Filter cache same as .94
> Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
> multivalued concept was changed from solr4 to solr6)
>
> *Please Help Note: I have given document cache 3 times memory for
> doc.cache.*
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>