[
https://issues.apache.org/jira/browse/SOLR-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-4356:
-------------------------------
Fix Version/s: (was: 4.7)
4.8
> SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs
> ---------------------------------------------------------------------
>
> Key: SOLR-4356
> URL: https://issues.apache.org/jira/browse/SOLR-4356
> Project: Solr
> Issue Type: Improvement
> Components: clients - java, Schema and Analysis, Tests
> Affects Versions: 4.1
> Environment: OS = Ubuntu 12.04
> Sun JAVA 7
> Max Java Heap Space = 2GB
> Apache Tomcat 7
> Hardware = {Intel core i3, 2GB RAM}
> Average no of fields in a Solr Doc = 100
> Reporter: Harish Verma
> Labels: performance, test
> Fix For: 4.8
>
> Attachments: memorydump1.png, memorydump2.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> we are testing solr 4.1 running inside tomcat 7 and java 7 with following
> options
> JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump"
> our source code looks like following:
> /**** START *****/
> int noOfSolrDocumentsInBatch = 0;
> for(int i=0 ; i<5000 ; i++) {
> SolrInputDocument solrInputDocument = getNextSolrInputDocument();
> server.add(solrInputDocument);
> noOfSolrDocumentsInBatch += 1;
> if(noOfSolrDocumentsInBatch == 10) {
> server.commit();
> noOfSolrDocumentsInBatch = 0;
> }
> }
> /**** END *****/
> the method "getNextSolrInputDocument()" generates a solr document with 100
> fields (average). Around 50 of the fields are of "text_general" type.
> Some of the "test_general" fields consist of approx 1000 words rest consists
> of few words. Ouf of total fields there are around 35-40 multivalued fields
> (not of type "text_general").
> We are indexing all the fields but storing only 8 fields. Out of these 8
> fields two are string type, five are long and one is boolean. So our index
> size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB.
> Why the memory is so high even though the index size is small?
> What is being stored in the memory? Our understanding is that after every
> commit documents are flushed to the disk.So nothing should remain in RAM
> after commit.
> We are using the following settings:
> server.commit() set waitForSearcher=true and waitForFlush=true
> solrConfig.xml has following properties set:
> directoryFactory = solr.MMapDirectoryFactory
> maxWarmingSearchers = 1
> text_general data type is being used as supplied in the schema.xml with the
> solr setup.
> maxIndexingThreads = 8(default)
> <autoCommit><maxTime>15000</maxTime><openSearcher>false</openSearcher></autoCommit>
> We get Java heap Out Of Memory Error after commiting around 3990 solr
> documents.Some of the snapshots of memory dump from profiler are attached.
> can somebody please suggest what should we do to minimize/optimize the memory
> consumption in our case with the reasons?
> also suggest what should be optimal values and reason for following
> parameters of solrConfig.xml
> useColdSearcher - true/false?
> maxwarmingsearchers- number
> spellcheck-on/off?
> omitNorms=true/false?
> omitTermFreqAndPositions?
> mergefactor? we are using default value 10
> java garbage collection tuning parameters ?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
