That first graph shows a JVM that does not have enough heap for the program it is running. Look at the bottom of the dips. That is the amount of memory still in use after a full GC.
You want those dips to drop to about half of the available heap, so I’d immediately increase that heap to 4G. That might not be enough, so you’ll need to to watch that graph after the increase. I’ve been using 8G heaps with Solr since version 1.2. We run this config with Java 8 on over 100 machines. We do not do any faceting, which can take more memory. SOLR_HEAP=8g # Use G1 GC -- wunder 2017-01-23 # Settings from https://wiki.apache.org/solr/ShawnHeisey GC_TUNE=" \ -XX:+UseG1GC \ -XX:+ParallelRefProcEnabled \ -XX:G1HeapRegionSize=8m \ -XX:MaxGCPauseMillis=200 \ -XX:+UseLargePages \ -XX:+AggressiveOpts \ " wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 27, 2020, at 12:48 AM, Jaan Arjasepp <jaan.arjas...@coop.ee> wrote: > > Hello, > > We have been using SOLR for quite some time. We used 6.0 and now we did a > little upgrade to our system and servers and we started to use 8.6.1. > We use it on a Windows Server 2019. > Java version is 11 > Basically using it in a default setting, except giving SOLR 2G of heap. It > used 512, but it ran out of memory and stopped responding. Not sure if it was > the issue. When older version, it managed fine with 512MB. > SOLR is not in a cloud mode, but in solo mode as we use it internally and it > does not have too many request nor indexing actually. > Document sizes are not big, I guess. We only use one core. > Document stats are here: > Num Docs: 3627341 > Max Doc: 4981019 > Heap Memory Usage: 434400 > Deleted Docs: 1353678 > Version: 15999036 > Segment Count: 30 > > The size of index is 2.66GB > > While making upgrade we had to modify one field and a bit of code that uses > it. Thats basically it. It works. > If needed more information about background of the system, I am happy to help. > > > But now to the issue I am having. > If SOLR is started, at first 40-60 minutes it works just fine. CPU is not > high, heap usage seem normal. All is good, but then suddenly, the heap usage > goes crazy, going up and down, up and down and CPU rises to 50-60% of the > usage. Also I noticed over the weekend, when there are no writing usage, the > CPU remains low and decent. I can try it this weekend again to see if and how > this works out. > Also it seems to me, that after 4-5 days of working like this, it stops > responding, but needs to be confirmed with more heap also. > > Heap memory usage via JMX and jconsole -> > https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing > As you can see, it starts of normal, but then goes crazy and it has been like > this over night. > > This is overall monitoring graphs, as you can see CPU is working hard or > hardly working. -> > https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing > VM summary can be found here -> > https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing > And finally to have better and quick overview of the SOLR executing > parameters that I have -> > https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing > > If you can point me what I have to do to make it work, then I appreciate it a > lot. > > Thank you in advance. > > Best regards, > Jaan > >
