Re: Solr 5.2.1 on Java 8 GC

On 4/28/2016 9:43 AM, Nick Vasilyev wrote: > I forgot to mention that the index is approximately 50 million docs split > across 4 shards (replication factor 2) on 2 solr replicas. Later in the thread, Jeff Wartes mentioned my wiki page for GC tuning.

Re: Solr 5.2.1 on Java 8 GC

How do you log GC frequency and time to compare it with other GC configurations? Also, do you tweak parameters automatically or is there a set of configuration that get tested? Lastly, I was under impression that G1 is not recommended to be used based on some issues with Lucene, so I haven't

Re: Solr 5.2.1 on Java 8 GC

On 30/04/16 17:34, Davis, Daniel (NIH/NLM) [C] wrote: > Bram, on the subject of brute force - if your script is "clever" and uses > binary first search, I'd love to adapt it to my environment. I am trying to > build a truly multi-tenant Solr because each of our indexes is tiny, but all >

RE: Solr 5.2.1 on Java 8 GC

to repeat this experiment, many, many times. From: Bram Van Dam [bram.van...@intix.eu] Sent: Saturday, April 30, 2016 7:10 AM To: solr-user@lucene.apache.org Subject: Re: Solr 5.2.1 on Java 8 GC On 29/04/16 16:40, Nick Vasilyev wrote: > Not sure if it

Re: Solr 5.2.1 on Java 8 GC

On 29/04/16 16:40, Nick Vasilyev wrote: > Not sure if it helps anyone, but I am seeing decent results with the > following. > > It was mostly a result of trial and error, I'm ashamed to admit that I've used a similar approach: wrote a simple test script to try out various GC settings with

Re: Solr 5.2.1 on Java 8 GC

Not sure if it helps anyone, but I am seeing decent results with the following. It was mostly a result of trial and error, I am not familiar with Java GC or even Java itself. I added my interpretation of what was happening, but I am not sure if it is right, take it for what it's worth. It'd be

Re: Solr 5.2.1 on Java 8 GC

Shawn Heisey’s page is the usual reference guide for GC settings: https://wiki.apache.org/solr/ShawnHeisey Most of the learnings from that are in the Solr 5.x startup scripts already, but your heap is bigger, so your mileage may vary. Some tools I’ve used while doing GC tuning: * VisualVM -

Re: Solr 5.2.1 on Java 8 GC

On Thu, Apr 28, 2016 at 12:21 PM, Nick Vasilyev wrote: > Hi Yonik, > > There are a lot of logistics involved with re-indexing and naturally > upgrading Solr. I was hoping that there is an easier alternative since this > is only a single back end script that is having

Re: Solr 5.2.1 on Java 8 GC

Hi Yonik, There are a lot of logistics involved with re-indexing and naturally upgrading Solr. I was hoping that there is an easier alternative since this is only a single back end script that is having problems. Is there any room for improvement with tweaking GC params? On Thu, Apr 28, 2016 at

Re: Solr 5.2.1 on Java 8 GC

Correction, the key_phrases is set up as follows: On Thu, Apr 28, 2016 at 12:03 PM, Nick Vasilyev wrote: > The working set is larger than the heap. This is our largest collection > and all shards combined

Re: Solr 5.2.1 on Java 8 GC

On Thu, Apr 28, 2016 at 11:50 AM, Nick Vasilyev wrote: > mmfr_exact is a string field. key_phrases is a multivalued string field. One guess is that top-level field caches (and UnInvertedField use) were removed in https://issues.apache.org/jira/browse/LUCENE-5666 While

Re: Solr 5.2.1 on Java 8 GC

The working set is larger than the heap. This is our largest collection and all shards combined would probably be around 60GB in total, there are also a few other much smaller collections. During normal operations the JVM memory utilization hangs between 17GB and 22GB if we aren't indexing any

Re: Solr 5.2.1 on Java 8 GC

32 GB is a pretty big heap. If the working set is really smaller than that, the extra heap just makes a full GC take longer. How much heap is used after a full GC? Take the largest value you see there, then add a bit more, maybe 25% more or 2 GB more. wunder Walter Underwood

Re: Solr 5.2.1 on Java 8 GC

mmfr_exact is a string field. key_phrases is a multivalued string field. On Thu, Apr 28, 2016 at 11:47 AM, Yonik Seeley wrote: > What about the field types though... are they single valued or multi > valued, string, text, numeric? > > -Yonik > > > On Thu, Apr 28, 2016 at

Re: Solr 5.2.1 on Java 8 GC

What about the field types though... are they single valued or multi valued, string, text, numeric? -Yonik On Thu, Apr 28, 2016 at 11:43 AM, Nick Vasilyev wrote: > Hi Yonik, > > I forgot to mention that the index is approximately 50 million docs split > across 4

Re: Solr 5.2.1 on Java 8 GC

Hi Yonik, I forgot to mention that the index is approximately 50 million docs split across 4 shards (replication factor 2) on 2 solr replicas. This particular script will filter items based on a category (10-~1,000,000 items in each) and run facets on top X terms for particular fields. Query

Re: Solr 5.2.1 on Java 8 GC

On Thu, Apr 28, 2016 at 11:29 AM, Nick Vasilyev wrote: > Hello, > > We recently upgraded to Solr 5.2.1 with jre1.8.0_74 and are seeing long GC > pauses when running jobs that do some hairy faceting. The same jobs worked > fine with our previous 4.6 Solr. What does a

Solr 5.2.1 on Java 8 GC

Hello, We recently upgraded to Solr 5.2.1 with jre1.8.0_74 and are seeing long GC pauses when running jobs that do some hairy faceting. The same jobs worked fine with our previous 4.6 Solr. The JVM is configured with 32GB heap with default GC settings, however I've been tweaking the GC settings