Thank you Shawn. Yes, I used the settings off of your site. I've
restarted the cluster and the CPU usage is back up again. Looking at it
now, it doesn't appear to be GC related.
Full log from one of the nodes that is pegging 13 CPU cores:
http://lovehorsepower.com/solr_gc.log.0.current
Thank you For the gceasy.io site - that is very slick! I'll use that in
the future. I can try using the standard settings, but again - at this
point it doesn't look GC related to me?
-Joe
On 2/12/2019 11:35 AM, Shawn Heisey wrote:
On 2/12/2019 7:35 AM, Joe Obernberger wrote:
Yesterday, we upgraded our 40 node cluster from solr 7.6.0 to solr
7.7.0. This morning, all the nodes are using 1200+% of CPU. It looks
like it's in garbage collection. We did reduce our HDFS cache size
from 11G to 6G, but other than that, no other parameters were changes.
Your message included a small excerpt from the GC log. That is not
helpful. We will need the entire GC log, possibly more than one log.
The log or logs should fully cover the timeframe where the problem
occurs. Full disclosure: Once obtained, I would use this website to
analyze GC log data:
http://gceasy.io
Parameters are:
GC_TUNE="-XX:+UseG1GC \
-XX:MaxDirectMemorySize=6g \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=16m \
-XX:MaxGCPauseMillis=300 \
-XX:InitiatingHeapOccupancyPercent=75 \
-XX:+UseLargePages \
-XX:ParallelGCThreads=16 \
-XX:-ResizePLAB \
-XX:+AggressiveOpts"
Looks like you've chosen to use G1 settings very similar to what I put
on my wiki page:
https://wiki.apache.org/solr/ShawnHeisey#Current_experiments
Those settings are not intended to be a canonical resource that
everyone can use. Your heap size is different than what I was using
when I worked on that, so you may need different settings.
Have you considered not using your own GC tuning, letting Solr's start
script handle that?
With the limited information available, my initial guess is that you
need a larger heap, that Java is spending all its time freeing up
enough memory to keep the program running.
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
https://www.avg.com
