Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-11-02 Thread Erick Erickson
What this sounds like is that somehow you were committing after every update in 8x but not in your 6x code. How that would have been change is anybody’s guess ;). It’s vaguely possible that your client is committing and you had IgnoreCommitOptimizeUpdateProcessorFactory defined in your update

RE: SOLR uses too much CPU and GC is also weird on Windows server

2020-11-02 Thread Jaan Arjasepp
Thanks for all for helping to think about it, but eventually found out that code was basically single record deleting/adding records. After it was batched up, then everything is back to normal. Funny thing is that 6.0.0. handled these requests somehow, but newer version did not. Anyway, we will

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-28 Thread Erick Erickson
DocValues=true are usually only used for “primitive” types, string, numerics, booleans and the like, specifically _not_ text-based. I say “usually” because there’s a special “SortableTextField” where it does make some sense to have a text-based field have docValues, but that’s intended for

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-28 Thread Walter Underwood
Double the heap. All that CPU is the GC trying to free up space. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 28, 2020, at 6:29 AM, Jaan Arjasepp wrote: > > Hi all, > > Its me again. Anyway, I did a little research and we tried different

RE: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-28 Thread Jaan Arjasepp
Hi all, Its me again. Anyway, I did a little research and we tried different things and well, some questions I want to ask and some things that I found. Well after monitoring my system with VirtualVM, I found that GC jumping is from 0.5GB to 2.5GB and it has 4GB of memory for now, so it should

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Walter Underwood
That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC. You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G.

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Emir Arnautović
Hi Jaan, You can also check in admin console in caches the sizes of field* caches. That will tell you if some field needs docValues=true. Regards, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 27

RE: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Jaan Arjasepp
Hi Erick, Thanks for this information, I will look into it. Main changes were regarding parsing the results JSON got from solr, not the queries or updates. Jaan P.S. configuration change about requestParser was not it. -Original Message- From: Erick Erickson Sent: 27 October 2020

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Erick Erickson
Jean: The basic search uses an “inverted index”, which is basically a list of terms and the documents they appear in, e.g. my - 1, 4, 9, 12 dog - 4, 8, 10 So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10. Makes it easy to search for my AND dog for instance,

RE: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Jaan Arjasepp
I found one little difference from old solrconfig and new one. It is in requestDispatchers section It does not have this, but we had this in old configuration. Maybe it helps, I will see. Jaan -Original Message- From: Jaan Arjasepp Sent: 27 October 2020 14:05 To:

RE: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Jaan Arjasepp
Hi Emir, I checked the solrconfig.xml file and we dont even use fieldValueCache. Also are you saying, I should check the schema and all the fields in the old solr and the new one to see if they match or contain similar settings? What does this uninverted value means? How to check this? As for

Re: SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Emir Arnautović
Hi Jaan, It can be several things: caches fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory. filterCache - you’ve changed setting for filter caches

SOLR uses too much CPU and GC is also weird on Windows server

2020-10-27 Thread Jaan Arjasepp
Hello, We have been using SOLR for quite some time. We used 6.0 and now we did a little upgrade to our system and servers and we started to use 8.6.1. We use it on a Windows Server 2019. Java version is 11 Basically using it in a default setting, except giving SOLR 2G of heap. It used 512, but