Hi, mike Is there any config option which let IndexReader to omit all norms and not to load norms to JVM heap?
Thanks & Best Regards! ------------------ Original ------------------ From: "lubin";<[email protected]>; Date: Sat, Sep 13, 2014 08:20 PM To: "java-user"<[email protected]>; Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer Hi, Mike In our use case, we have thousands of index fields, different kind of document have different fields. Do you meant that norms field will consume large memory? Why? If we decide to disable norms, do we need to rebuild our index entirely? By the way, We have 8 million documents and our jvm heap is 5G. Thanks & Best Regards! ------------------ Original ------------------ From: "Michael McCandless";<[email protected]>; Date: Sat, Sep 13, 2014 06:29 PM To: "Lucene Users"<[email protected]>; Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer The warmer just tries to load norms/docValues/etc. for all fields that have them enabled ... so this is likely telling you an IndexReader would also hit OOME. You either need to reduce the number of fields you have indexed, or at least disable norms (takes 1 byte per doc per indexed field regardless of whether that doc had indexed that field), or increase HEAP to the JVM. Mike McCandless http://blog.mikemccandless.com On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <[email protected]> wrote: > Hi, all > we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use > lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could > any body give me a hand? Strack trace is as follows: > > > > > > > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518) > > > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228) > at > org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188) > at > org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166) > at > org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519) > at > org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) > > > > > Thanks & Best Regards! --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
