Hi Erick,
This is one of the errors I get (at the 4GB memory machine) and after
a while Tomcat crashes:
SEVERE: SolrIndexWriter was not closed prior to finalize(),
indicates a bug -- POSSIBLE RESOURCE LEAK!!!
And this is part of my solrconfig.xml (I'm indexing 200k documents per run):
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>128</ramBufferSizeMB>
<maxFieldLength>1024000</maxFieldLength>
<writeLockTimeout>60000</writeLockTimeout>
<commitLockTimeout>60000</commitLockTimeout>
<lockType>native</lockType>
</indexDefaults>
<mainIndex>
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>32</ramBufferSizeMB>
<mergeFactor>10</mergeFactor>
<unlockOnStartup>false</unlockOnStartup>
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="maxCommitsToKeep">1</str>
<str name="maxOptimizedCommitsToKeep">0</str>
</deletionPolicy>
<infoStream file="INFOSTREAM.txt">false</infoStream>
</mainIndex>
<updateHandler class="solr.DirectUpdateHandler2">
<autoCommit>
<maxTime>60000</maxTime>
</autoCommit>
</updateHandler>
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="0"
initialSize="0"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="0"
initialSize="0"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="0"
initialSize="0"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">static firstSearcher warming in solrconfig.xml</str>
</lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
In my opinion Solr has to work fine with this settings. Any comments?
Thanks again.
Greetings,
Marc
2011/8/31 Erick Erickson <erickerick...@gmail.com>
>
> See solrconfig.xml, particularly ramBufferSizeMB,
> also maxBufferedDocs.
>
> There's no reason you can't index as many documents
> as you want, unless your documents are absolutely
> huge (as in 100s of M, possibly G size).
>
> Are you actually getting out of memory problems?
>
> Erick
>
> On Tue, Aug 30, 2011 at 4:24 PM, Marc Jacobs <jacob...@gmail.com> wrote:
> > Hi Chris,
> >
> > Thanks for the response.
> > Eventualy I want to install Solr on a machine with a maximum memory of 4GB.
> > I tried to index the data on that machine before, but it resulted in index
> > locks and memory errors.
> > Is 4GB not enough to index 100,000 documents in a row? How much should it
> > be? Is there a way to tune this?
> >
> > Regards,
> >
> > Marc
> >
> > 2011/8/30 Chris Hostetter <hossman_luc...@fucit.org>
> >
> >>
> >> : The current system I'm using has 150GB of memory and while I'm indexing
> >> the
> >> : memoryconsumption is growing and growing (eventually more then 50GB).
> >> : In the attached graph (http://postimage.org/image/acyv7kec/) I indexed
> >> about
> >> : 70k of office-documents (pdf,doc,xls etc) and between 1 and 2 percent
> >> throws
> >>
> >> Unless i'm missunderstanding sometihng about your graph, only ~12GB of
> >> memory is used by applications on that machine. About 60GB is in use by
> >> the "filesystem cache".
> >>
> >> The Filesystem cache is not memory being used by Solr, it's memory that is
> >> free and not in use by an application, so your OS is (wisely) using it to
> >> cache files from disk that you've recently accessed in case you need them
> >> again. This is handy, and for max efficients (when keeping your index on
> >> disk) it's useful to make sure you allocate resources so that you have
> >> enough extra memory on your server that the entire index can be kept in
> >> the filesystem cache -- but the OS will happily free up that space for
> >> other apps that need it if they ask for more memory.
> >>
> >> : After indexing the memoryconsumption isn't dropping. Even after an
> >> optimize
> >> : command it's still there.
> >>
> >> as for why your "Used" memory grows to ~12GB and doesn't decrease even
> >> after an optimize: that's the way the Java memory model works. whe nyou
> >> run the JVM you specificy (either explicitly or implicitly via defaults) a
> >> min & max heap size for hte JVM to allocate for itself. it starts out
> >> asking the OS for the min, and as it needs more it asks for more up to the
> >> max. but (most JVM implementations i know of) don't "give back" ram to
> >> the OS if they don't need it anymore -- they keep it as free space in the
> >> heap for future object allocation.
> >>
> >>
> >>
> >> -Hoss
> >>
> >
