On 5/6/2015 8:55 AM, adfel70 wrote: > Thank you for the detailed answer. > How can I decrease the impact of opening a searcher in such a large index? > especially the impact of heap usage that causes OOM.
See the wiki link I sent. It talks about some of the things that require a lot of heap and ways you can reduce those requirements. The lists are nowhere near complete. http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > regarding GC tuning - I am doint that. > here are the params I use: > AggresiveOpts > UseLargePages > ParallelRefProcEnabled > CMSParallelRemarkEnabled > CMSMaxAbortablePrecleanTime=6000 > CMDTriggerPermRatio=80 > CMSInitiatingOccupancyFraction=70 > UseCMSInitiatinOccupancyOnly > CMSFullGCsBeforeCompaction=1 > PretenureSizeThreshold=64m > CMSScavengeBeforeRemark > UseConcMarkSweepGC > MaxTenuringThreshold=8 > TargetSurvivorRatio=90 > SurviorRatio=4 > NewRatio=2 > Xms16gb > Xmn28gb This list seems to have come from re-typing the GC options. If this is a cut/paste, I would not expect it to work -- there are typos and part of each option is missing other characters. Assuming that this is not cut/paste, it is mostly similar to the CMS options that I once used for my own index: http://wiki.apache.org/solr/ShawnHeisey#CMS_.28ConcurrentMarkSweep.29_Collector > How many documents per shard are recommended? > Note that I use nested documents. total collection size is 3 billion docs, > number of parent docs is 600 million. the rest are children. For the G1 collector, you'd want to limit each shard to about 100 million docs. I have no idea about limitations and capabilities where very large memory allocations are concerned with the CMS collector. Running the latest Java 8 is *strongly* recommended, no matter what collector you're using, because recent versions have incorporated GC improvements with large memory allocations. With Java 8u40 and later, the limitations for 16MB huge allocations on the G1 collector might not even apply. Thanks, Shawn
