On 9/3/2021 8:34 AM, Dominic Humphries wrote:
Thanks for replying!
it is indeed the same Java (OpenJDK 11) we're using for both. I tried
upgrading to 14 and the heap *does* stay a little smaller, but we still see
the large (>10sec) GC durations happening.
I've not been able to get switching to CMS to work, unfortunately
What did you see with CMS?
One thing that I would try, if you haven't already, and assuming you
have enough memory installed, is to increase the max heap size. If the
available headroom is high enough, Java will be far more likely to use
the generation-specific collectors, which are much faster than a full
GC, and some phases of those collectors are concurrent, meaning they do
not pause the app (Solr in this case) while they run.
If you are running at least version 11u9, then you have access to a new
collector that hasn't received a lot of testing from us yet --
Shenandoah. I enabled Shenandoah on my own Solr install just now ... I
had thought it required a newer Java version than 11, so I hadn't tried
it yet. This is what I have in /etc/default/solr.in.sh :
GC_TUNE=" \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
-XX:+UseShenandoahGC \
-XX:+ParallelRefProcEnabled \
-XX:ParallelGCThreads=6 \
"
My server is NUMA hardware, which is why I included that option. I have
12 CPU cores, so I told Java to use 6 threads for GC. It is probably
unnecessary to include ParallelRefProcEnabled but I like to be explicit
in case defaults change in later versions of Java.
I would recommend trying Shenandoah if you have a new enough version of
OpenJDK. It's supposed to be the best low-pause option currently
available. And consider making your heap larger, if you have enough
memory. Make sure there's still enough un-allocated memory available
for caching purposes -- you don't want to use all the memory in the
machine for Solr's heap.
If my install ever gets around to doing GC (its heap is a lot bigger
than it really needs to be) then I might know whether Shenandoah is
beneficial.
Thanks,
Shawn
