Thanks for your reply.
I set autowarmcount = 0 for both LRUCache and the queryCache but still I got
these errors on heavy reindexing (4M docs as fast as possible each doc <
10K). I removed firstSearcher and newSearcher but I still got the same
errors.
The strange thing is that now when the server have returned to delta
indexing state I can survive with 256M memory where I before died on 512M.
So the conclusion is perhaps if I reduce the number of documents indexed per
second ( by introducing Thread.sleep(X) ) then I will survive with lower
heap ? Seems like wrong assumption but...
Do lucene never use disksorting ? I mean something like this:
if(estimateRamUsed(resultset)) > maxRAMPerThread)
{
doDiskSort(resultset)
}
else....
On 1/22/08, Mike Klaas <[EMAIL PROTECTED]> wrote:
>
> Queries involving sorting can occupy a lot of memory. During
> autowarming you need 2x peak memory usage. The only thing you can do
> is increase your max heap size or be careful about cache autowarming
> (possibly turning it off).
>
> cheers,
> -Mike
>
> On 21-Jan-08, at 9:44 PM, Marcus Herou wrote:
>
> > Hi.
> >
> > I get OOE with Solr 1.3 Autowarm seem to be the villain in
> > cojunction with FieldCache somehow.
> > JVM args: -Xmx512m -Xms512m -Xss128k
> >
> > Index size is ~4 Million docs, where I index text and store
> > database primary keys.
> > du /srv/solr/feedItem/data/index/
> > 1.7G /srv/solr/feedItem/data/index/
> >
> > To ensure that the docs I index do not swell to much I only allow
> > 5K per doc to over the wire i.e. I substring 0, 5000 on the field
> > "content"
> >
> > I have removed "firstSearcher" and "newSearcher" since the queries
> > I used before killed performance on reindexing the whole index. I
> > will add them later again when I get into a delta update index state.
> >
> > Stacktrace.
> > [06:25:53.122] [null] /update wt=xml&version=2.2 0 3165
> > [06:25:53.877] Error during auto-warming of
> > key:[EMAIL PROTECTED]:java.lang.OutOfMemo
> > ryError: Java heap space
> > [06:25:53.877] at org.apache.lucene.index.TermBuffer.toTerm
> > (TermBuffer.java:104)
> > [06:25:53.877] at org.apache.lucene.index.SegmentTermEnum.term
> > (SegmentTermEnum.java:159)
> > [06:25:53.877] at org.apache.lucene.index.SegmentMergeInfo.next
> > (SegmentMergeInfo.java:66)
> > [06:25:53.877] at org.apache.lucene.index.MultiTermEnum.next
> > (MultiReader.java:315)
> > [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl
> > $10.createValue(FieldCacheImpl.java:388)
> > [06:25: 53.877] at org.apache.lucene.search.FieldCacheImpl
> > $Cache.get(FieldCacheImpl.java:72)
> > [06:25:53.877] at
> > org.apache.lucene.search.FieldCacheImpl.getStringIndex
> > (FieldCacheImpl.java:350)
> > [06:25:53.877] at
> > org.apache.lucene.search.FieldSortedHitQueue.comparatorString
> > (FieldSortedHitQueue.java:266)
> > [06:25:53.877] at org.apache.lucene.search.FieldSortedHitQueue
> > $1.createValue(FieldSortedHitQueue.java:182)
> > [06:25:53.877] at org.apache.lucene.search.FieldCacheImpl$Cache.get
> > (FieldCacheImpl.java :72)
> > [06:25:53.877] at
> > org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator
> > (FieldSortedHitQueue.java:155)
> > [06:25:53.877] at
> > org.apache.lucene.search.FieldSortedHitQueue.<init>
> > (FieldSortedHitQueue.java :56)
> > [06:25:53.877] at
> > org.apache.solr.search.SolrIndexSearcher.getDocListNC
> > (SolrIndexSearcher.java:862)
> > [06:25:53.877] at
> > org.apache.solr.search.SolrIndexSearcher.getDocListC
> > (SolrIndexSearcher.java:808)
> > [06:25: 53.877] at org.apache.solr.search.SolrIndexSearcher.access
> > $000(SolrIndexSearcher.java:56)
> > [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher
> > $2.regenerateItem(SolrIndexSearcher.java:254)
> > [06:25:53.877] at org.apache.solr.search.LRUCache.warm
> > (LRUCache.java:192)
> > [06:25:53.877] at org.apache.solr.search.SolrIndexSearcher.warm
> > (SolrIndexSearcher.java:1393)
> > [06:25:53.877] at org.apache.solr.core.SolrCore$2.call
> > (SolrCore.java :702)
> > [06:25:53.877] at java.util.concurrent.FutureTask$Sync.innerRun
> > (FutureTask.java:269)
> > [06:25:53.877] at java.util.concurrent.FutureTask.run
> > (FutureTask.java:123)
> > [06:25:53.877] at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask (ThreadPoolExecutor.java:650)
> > [06:25:53.877] at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:675)
> > [06:25:53.877] at java.lang.Thread.run(Thread.java:595)
> >
> > Help anyone?
> >
> > Attaching schema.xml and solrconfig.xml
> >
> > Kindly
> >
> > //Marcus Herou
> >
> > <schema.xml><solrconfig.xml>
>
>
--
Marcus Herou Solution Architect and Java developer Tailsweep AB
+46702561312
[EMAIL PROTECTED]
http://www.tailsweep.com/
http://blogg.tailsweep.com/
