Yes, use a bigger heap and the generational collector. With CMS, we had an 8G heap with 2G in the short-life generation. Nearly all of the allocations for handling a search request are free at the end of that request, so Solr can make use of a LOT of new generation space.
With Solr 8.6, Java 11, and G1, we’ve moved to 16 G for all JVMs. We had some excessive GC problems at 8G. Our clusters don’t do facets, but they do some heavy searching. Our biggest cluster is 60 Mdocs, 8 shards, average query length is 25 terms. We use 36 CPU EC2 instances. -Xms16g -Xmx16g -Xss256k -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -XX:+UseG1GC -XX:+UseLargePages -XX:-OmitStackTraceInFastThrow -XX:MaxGCPauseMillis=250 wunder Walter Underwood [email protected] http://observer.wunderwood.org/ (my blog) > On Sep 3, 2021, at 9:08 AM, Shawn Heisey <[email protected]> wrote: > > On 9/3/2021 8:34 AM, Dominic Humphries wrote: >> Thanks for replying! >> >> it is indeed the same Java (OpenJDK 11) we're using for both. I tried >> upgrading to 14 and the heap *does* stay a little smaller, but we still see >> the large (>10sec) GC durations happening. >> >> I've not been able to get switching to CMS to work, unfortunately > > > What did you see with CMS? > > One thing that I would try, if you haven't already, and assuming you have > enough memory installed, is to increase the max heap size. If the available > headroom is high enough, Java will be far more likely to use the > generation-specific collectors, which are much faster than a full GC, and > some phases of those collectors are concurrent, meaning they do not pause the > app (Solr in this case) while they run. > > If you are running at least version 11u9, then you have access to a new > collector that hasn't received a lot of testing from us yet -- Shenandoah. I > enabled Shenandoah on my own Solr install just now ... I had thought it > required a newer Java version than 11, so I hadn't tried it yet. This is > what I have in /etc/default/solr.in.sh : > > GC_TUNE=" \ > -XX:+AlwaysPreTouch \ > -XX:+UseNUMA \ > -XX:+UseShenandoahGC \ > -XX:+ParallelRefProcEnabled \ > -XX:ParallelGCThreads=6 \ > " > > My server is NUMA hardware, which is why I included that option. I have 12 > CPU cores, so I told Java to use 6 threads for GC. It is probably > unnecessary to include ParallelRefProcEnabled but I like to be explicit in > case defaults change in later versions of Java. > > I would recommend trying Shenandoah if you have a new enough version of > OpenJDK. It's supposed to be the best low-pause option currently available. > And consider making your heap larger, if you have enough memory. Make sure > there's still enough un-allocated memory available for caching purposes -- > you don't want to use all the memory in the machine for Solr's heap. > > If my install ever gets around to doing GC (its heap is a lot bigger than it > really needs to be) then I might know whether Shenandoah is beneficial. > > Thanks, > Shawn >
