Re: Long STW GCs with Solr Cloud

2016-06-17 Thread Walter Underwood
I try to adjust the new generation size so that it can handle all the allocations needed for HTTP requests. Those short-lived objects should never come from tenured space. Even without facets, I run a pretty big new generation, 2 GB in an 8 GB heap. The tenured space will always grow in Solr,

Re: Long STW GCs with Solr Cloud

2016-06-17 Thread Jeff Wartes
For what it’s worth, I looked into reducing the allocation footprint of CollapsingQParserPlugin a bit, but without success. See https://issues.apache.org/jira/browse/SOLR-9125 As it happened, I was collapsing on a field with such high cardinality that the chances of a query even doing much

Re: Long STW GCs with Solr Cloud

2016-06-17 Thread Ere Maijala
17.6.2016, 11.05, Bernd Fehling kirjoitti: Am 17.06.2016 um 09:06 schrieb Ere Maijala: 16.6.2016, 1.41, Shawn Heisey kirjoitti: If you want to continue avoiding G1, you should definitely be using CMS. My recommendation right now would be to try the G1 settings on my wiki page under the

Re: Long STW GCs with Solr Cloud

2016-06-17 Thread Bernd Fehling
Am 17.06.2016 um 09:06 schrieb Ere Maijala: > 16.6.2016, 1.41, Shawn Heisey kirjoitti: >> If you want to continue avoiding G1, you should definitely be using >> CMS. My recommendation right now would be to try the G1 settings on my >> wiki page under the heading "Current experiments" or the CMS

Re: Long STW GCs with Solr Cloud

2016-06-17 Thread Ere Maijala
16.6.2016, 1.41, Shawn Heisey kirjoitti: If you want to continue avoiding G1, you should definitely be using CMS. My recommendation right now would be to try the G1 settings on my wiki page under the heading "Current experiments" or the CMS settings just below that. For what it's worth, we're

Re: Long STW GCs with Solr Cloud

2016-06-16 Thread Cas Rusnov
Hey thanks for your reply. Looks like running the suggested CMS config from Shawn, we're getting some nodes with 30+sec pauses, I gather due to large heap, interestingly enough while the scenario Jeff talked about is remarkably similar (we use field collapsing), including the performance aspects

Re: Long STW GCs with Solr Cloud

2016-06-16 Thread Jeff Wartes
Check your gc log for CMS “concurrent mode failure” messages. If a concurrent CMS collection fails, it does a stop-the-world pause while it cleans up using a *single thread*. This means the stop-the-world CMS collection in the failure case is typically several times slower than a concurrent

Re: Long STW GCs with Solr Cloud

2016-06-15 Thread Cas Rusnov
Hey Shawn! Thanks for replying. Yes I meant HugePages not HugeTable, brain fart. I will give the transparent off option a go. I have attempted to use your CMS configs as is and also the default settings and the cluster dies under our load (basically a node will get a 35-60s GC STW and then the

Re: Long STW GCs with Solr Cloud

2016-06-15 Thread Shawn Heisey
On 6/15/2016 3:05 PM, Cas Rusnov wrote: > After trying many of the off the shelf configurations (including CMS > configurations but excluding G1GC, which we're still taking the > warnings about seriously), numerous tweaks, rumors, various instance > sizes, and all the rest, most of which

Long STW GCs with Solr Cloud

2016-06-15 Thread Cas Rusnov
I know this has been discussed in the past (although not too recently), but the advices in those have failed us, so here we are. Some basics: We're running Solr 6 (6.0.0 48c80f91b8e5cd9b3a9b48e6184bd53e7619e7e3 - nknize - 2016-04-01 14:41:49), on Java 8 (OpenJDK Runtime Environment (build