Thanks a lot for the responses, after the optimize is complete and i have some time to experiment ill throw some of these settings in place,
On Tue, Jul 25, 2017 at 4:39 PM, Walter Underwood <wun...@wunderwood.org> wrote: > I’ve never been fond of elaborate GC settings. I prefer to set a few > things then let it run. I know someone who’s worked on garbage collectors > for thirty years. I don’t second guess him. > > From watching GC performance under a load benchmark (CMS/ParNew) with Solr > 4.x, I increased the new space. Individual requests make a lot of > allocations that are garbage at the end of the request. All of those need > to come from the new space. If new space is not big enough, they’ll be > allocated from tenured space. I settled on an 8G heap with 2G of new space. > These are the options (Java 7): > > export CATALINA_OPTS="$CATALINA_OPTS -d64" > export CATALINA_OPTS="$CATALINA_OPTS -server" > export CATALINA_OPTS="$CATALINA_OPTS -Xms8g" > export CATALINA_OPTS="$CATALINA_OPTS -Xmx8g" > export CATALINA_OPTS="$CATALINA_OPTS -XX:NewSize=2g" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParNewGC" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+ExplicitGCInvokesConcurrent" > export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+PrintGCDetails" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+PrintGCTimeStamps" > export CATALINA_OPTS="$CATALINA_OPTS -XX:-TraceClassUnloading" > export CATALINA_OPTS="$CATALINA_OPTS -Xloggc:$CATALINA_HOME/logs/gc.log" > export CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError" > export CATALINA_OPTS="$CATALINA_OPTS -XX:HeapDumpPath=$CATALINA_ > HOME/logs/“ > > Tenured space will rise slowly, mostly because of cache ejections and > background merges, I think. Cache ejections from an LRU cache will almost > always be in tenured space. > > For Java 8 and Solr 6.5.1, we are running the G1 collector and very happy > with it. We run the options recommended by Shawn Heisey on this list. > > SOLR_HEAP=8g > # Use G1 GC -- wunder 2017-01-23 > # Settings from https://wiki.apache.org/solr/ShawnHeisey > GC_TUNE=" \ > -XX:+UseG1GC \ > -XX:+ParallelRefProcEnabled \ > -XX:G1HeapRegionSize=8m \ > -XX:MaxGCPauseMillis=200 \ > -XX:+UseLargePages \ > -XX:+AggressiveOpts \ > “ > > Last week, I benchmarked the 4.x config handling 15,000 requests/minute > with a 95th percentile response time of 30 ms, using production logs. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > > On Jul 25, 2017, at 1:24 PM, Markus Jelsma <markus.jel...@openindex.io> > wrote: > > > > Upgrade to 6.x and get, in general, decent JVM settings. And decrease > your heap, having it so extremely large is detrimental at best. > > > > Our shards can be 25 GB in size, but we run fine (apart from other > problems recently discovered) with a 900 MB heap, so you probably have a > lot of room to spare. Your max heap is over a 100 times larger than ours, > your index just around 16 times. It should work with less. > > > > As a bonus, with a smaller heap, you can have much more index data in > mapped memory. > > > > Regards, > > Markus > > > > -----Original message----- > >> From:David Hastings <hastings.recurs...@gmail.com> > >> Sent: Tuesday 25th July 2017 22:15 > >> To: solr-user@lucene.apache.org > >> Subject: Re: Optimize stalls at the same point > >> > >> it turned out that i think it was a large GC operation, as it has since > >> resumed optimizing. current java options are as follows for the > indexing > >> server (they are different for the search servers) if you have any > >> suggestions as to changes I am more than happy to hear them, honestly > they > >> have just been passed down from one installation to the next ever since > we > >> used to use tomcat to host solr > >> -server -Xss256k -d64 -Xmx100000m -Xms7000m-XX:NewRatio=3 > >> -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 > -XX:MaxTenuringThreshold=8 > >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 > >> -XX:ParallelGCThreads=8 -XX:+CMSScavengeBeforeRemark > >> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly > >> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime= > 6000 > >> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc > >> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps > >> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution > >> -XX:+PrintGCApplicationStoppedTime > >> -Xloggc:XXXXXX/solr-5.2.1/server/logs/solr_gc.log > >> > >> and for my live searchers i use: > >> server Xss256k Xms50000m Xmx50000m XX:NewRatio=3 XX:SurvivorRatio=4 > >> XX:TargetSurvivorRatio=90 XX:MaxTenuringThreshold=8 > XX:+UseConcMarkSweepGC > >> XX:+UseParNewGC XX:ConcGCThreads=4 XX:ParallelGCThreads=8 > >> XX:+CMSScavengeBeforeRemark XX:PretenureSizeThreshold=64m > >> XX:+UseCMSInitiatingOccupancyOnly XX:CMSInitiatingOccupancyFraction=50 > >> XX:CMSMaxAbortablePrecleanTime=6000 XX:+CMSParallelRemarkEnabled > >> XX:+ParallelRefProcEnabled verbose:gc XX:+PrintHeapAtGC > XX:+PrintGCDetails > >> XX:+PrintGCDateStamps XX:+PrintGCTimeStamps > XX:+PrintTenuringDistribution > >> XX:+PrintGCApplicationStoppedTime Xloggc:/SSD2TB01/solr > >> 5.2.1/server/logs/solr_gc.log > >> > >> > >> > >> On Tue, Jul 25, 2017 at 4:02 PM, Walter Underwood < > wun...@wunderwood.org> > >> wrote: > >> > >>> Are you sure you need a 100GB heap? The stall could be a major GC. > >>> > >>> We run with an 8GB heap. We also run with Xmx equal to Xms, growing > memory > >>> to the max was really time-consuming after startup. > >>> > >>> What version of Java? What GC options? > >>> > >>> wunder > >>> Walter Underwood > >>> wun...@wunderwood.org > >>> http://observer.wunderwood.org/ (my blog) > >>> > >>> > >>>> On Jul 25, 2017, at 12:03 PM, David Hastings < > >>> hastings.recurs...@gmail.com> wrote: > >>>> > >>>> I am trying to optimize a rather large index (417gb) because its > sitting > >>> at > >>>> 28% deletions. However when optimizing, it stops at exactly 492.24 GB > >>>> every time. When I restart solr it will fall back down to 417 gb, and > >>>> again, if i send an optimize command, the exact same 492.24 GB and it > >>> stops > >>>> optimizing. There is plenty of space on the drive, and im running it > >>>> at -Xmx100000m -Xms7000m on a machine with 132gb of ram and 24 > cores. I > >>>> have never ran into this problem before but also never had the index > get > >>>> this large. Any ideas? > >>>> (solr 5.2 btw) > >>>> thanks, > >>>> -Dave > >>> > >>> > >> > >
