I agree, memory profiler or heap dump or small test case is the next
step... the code looks fine.
This is always a single thread adding docs?
Are you really certain that the iterator only iterates over 2500 docs?
What analyzer are you using?
Mike
On Thu, Mar 4, 2010 at 4:50 AM, Ian Lea <ian....@gmail.com> wrote:
> Have you run it through a memory profiler yet? Seems the obvious next step.
>
> If that doesn't help, cut it down to the simplest possible
> self-contained program that demonstrates the problem and post it here.
>
>
> --
> Ian.
>
>
> On Thu, Mar 4, 2010 at 6:04 AM, ajay_gupta <ajay...@gmail.com> wrote:
>>
>> Erick,
>> w_context and context_str are local to this method and are used only for
>> 2500 K documents not entire 70 k. I am clearing the hashmap after each 2500k
>> doc processing and also I printed memory consumed by hashmap which is kind
>> of constant for each chunk processing. For each invocation of
>> update_context memory should be kind of constant but after each invocation
>> it increase few MB's and after 70k it goes OOM so something wrong is
>> happening inside update_context some operation like search/update/add
>> document is creating some memory and which is not release after returning
>> from this method.
>>
>> -Ajay
>>
>>
>> Erick Erickson wrote:
>>>
>>> The first place I'd look is how big my your strings
>>> got. w_context and context_str come to mind. My
>>> first suspicion is that you're building ever-longer
>>> strings and around 70K documents your strings
>>> are large enough to produce OOMs.
>>>
>>> FWIW
>>> Erick
>>>
>>> On Wed, Mar 3, 2010 at 1:09 PM, ajay_gupta <ajay...@gmail.com> wrote:
>>>
>>>>
>>>> Mike,
>>>> Actually my documents are very small in size. We have csv files where
>>>> each
>>>> record represents a document which is not very large so I don't think
>>>> document size is an issue.
>>>> For each record I am tokenizing it and for each token I am keeping 3
>>>> neighbouring tokens in a Hashtable. After X number of documents where X
>>>> is
>>>> currently 2500 I am creating
>>>> index by following code:
>>>>
>>>> //Initialization step done only at
>>>> starting
>>>>
>>>> cram = FSDirectory.open(new
>>>> File("lucenetemp2"));
>>>> context_writer = new IndexWriter(cram,
>>>> analyzer, true,
>>>> IndexWriter.MaxFieldLength.LIMITED);
>>>>
>>>> // After each 2500 docs
>>>>
>>>> update_context()
>>>> {
>>>> context_writer.commit();
>>>> context_writer.optimize();
>>>>
>>>> IndexSearcher is = new IndexSearcher(cram);
>>>> IndexReader ir = is.getIndexReader();
>>>> Iterator<String> it = context.keySet().iterator();
>>>>
>>>> while(it.hasNext())
>>>> {
>>>> String word = it.next();
>>>> // This is all the context of "word" for
>>>> all
>>>> the 2500 docs
>>>> StringBuffer w_context =
>>>> context.get(word);
>>>> Term t = new Term("Word", word);
>>>> TermQuery tq = new TermQuery(t);
>>>> TopScoreDocCollector collector =
>>>> TopScoreDocCollector.create(1, false);
>>>> is.search(tq,collector);
>>>> ScoreDoc[] hits =
>>>> collector.topDocs().scoreDocs;
>>>>
>>>> if(hits.length!=0)
>>>> {
>>>> int id = hits[0].doc;
>>>> TermFreqVector tfv =
>>>> ir.getTermFreqVector(id, "Context");
>>>>
>>>> // This creates context string
>>>> from
>>>> TermFreqVector. For e.g if
>>>> TermFreqVector is word1(2), word2(1),word3(2) then its output is
>>>> // context_str="word1 word1 word2
>>>> word3 word3"
>>>> String context_str =
>>>> getContextString(tfv);
>>>>
>>>>
>>>> w_context.append(context_str);
>>>> Document new_doc = new Document();
>>>> new_doc.add(new Field("Word",
>>>> word,Field.Store.YES,
>>>> Field.Index.NOT_ANALYZED));
>>>> new_doc.add(new Field("Context",
>>>> w_context.toString(),Field.Store.YES,
>>>> Field.Index.ANALYZED, Field.TermVector.YES));
>>>> context_writer.updateDocument(t,
>>>> new_doc);
>>>>
>>>> }else{
>>>>
>>>> Document new_doc = new Document();
>>>> new_doc.add(new Field("Word",
>>>> word,Field.Store.YES,
>>>> Field.Index.NOT_ANALYZED));
>>>> new_doc.add(new Field("Context",
>>>> w_context.toString(),Field.Store.YES,
>>>> Field.Index.ANALYZED, Field.TermVector.YES));
>>>>
>>>> context_writer.addDocument(new_doc);
>>>>
>>>> }
>>>> }
>>>> ir.close();
>>>> is.close();
>>>>
>>>> }
>>>>
>>>>
>>>> I am printing memory also after each invocation of this method and I
>>>> observed that after each call of update_context memory increases and when
>>>> it
>>>> reaches around 65-70k it goes outofmemory so somewhere memory is
>>>> increasing
>>>> in each invocation. I thought each invocation should take constant amount
>>>> of
>>>> memory and it should not be increased cumulatively. Also after each
>>>> invocation of Update_context I am also calling System.gc() to release
>>>> memory
>>>> and I also tried various other parameters like
>>>> context_writer.setMaxBufferedDocs()
>>>> context_writer.setMaxMergeDocs()
>>>> context_writer.setRAMBufferSizeMB()
>>>> I set these parameters smaller values as well but nothing worked.
>>>>
>>>> Any hint will be very helpful.
>>>>
>>>> Thanks
>>>> Ajay
>>>>
>>>>
>>>> Michael McCandless-2 wrote:
>>>> >
>>>> > The worst case RAM usage for Lucene is a single doc with many unique
>>>> > terms. Lucene allocates ~60 bytes per unique term (plus space to hold
>>>> > that term's characters = 2 bytes per char). And, Lucene cannot flush
>>>> > within one document -- it must flush after the doc has been fully
>>>> > indexed.
>>>> >
>>>> > This past thread (also from Paul) delves into some of the details:
>>>> >
>>>> > http://lucene.markmail.org/thread/pbeidtepentm6mdn
>>>> >
>>>> > But it's not clear whether that is the issue affecting Ajay -- I think
>>>> > more details about the docs, or, some code fragments, could help shed
>>>> > light.
>>>> >
>>>> > Mike
>>>> >
>>>> > On Tue, Mar 2, 2010 at 8:47 AM, Murdoch, Paul <paul.b.murd...@saic.com>
>>>> > wrote:
>>>> >> Ajay,
>>>> >>
>>>> >> Here is another thread I started on the same issue.
>>>> >>
>>>> >>
>>>> http://stackoverflow.com/questions/1362460/why-does-lucene-cause-oom-whe
>>>> >> n-indexing-large-files
>>>> >>
>>>> >> Paul
>>>> >>
>>>> >>
>>>> >> -----Original Message-----
>>>> >> From: java-user-return-45254-paul.b.murdoch=saic....@lucene.apache.org
>>>> >> [mailto:java-user-return-45254-PAUL.B.MURDOCH=saic.com@
>>>> lucene.apache.org
>>>> >> ] On Behalf Of ajay_gupta
>>>> >> Sent: Tuesday, March 02, 2010 8:28 AM
>>>> >> To: java-user@lucene.apache.org
>>>> >> Subject: Lucene Indexing out of memory
>>>> >>
>>>> >>
>>>> >> Hi,
>>>> >> It might be general question though but I couldn't find the answer
>>>> yet.
>>>> >> I
>>>> >> have around 90k documents sizing around 350 MB. Each document contains
>>>> a
>>>> >> record which has some text content. For each word in this text I want
>>>> to
>>>> >> store context for that word and index it so I am reading each document
>>>> >> and
>>>> >> for each word in that document I am appending fixed number of
>>>> >> surrounding
>>>> >> words. To do that first I search in existing indices if this word
>>>> >> already
>>>> >> exist and if it is then I get the content and append the new context
>>>> and
>>>> >> update the document. In case no context exist I create a document with
>>>> >> fields "word" and "context" and add these two fields with values as
>>>> word
>>>> >> value and context value.
>>>> >>
>>>> >> I tried this in RAM but after certain no of docs it gave out of memory
>>>> >> error
>>>> >> so I thought to use FSDirectory method but surprisingly after 70k
>>>> >> documents
>>>> >> it also gave OOM error. I have enough disk space but still I am
>>>> getting
>>>> >> this
>>>> >> error.I am not sure even for disk based indexing why its giving this
>>>> >> error.
>>>> >> I thought disk based indexing will be slow but atleast it will be
>>>> >> scalable.
>>>> >> Could someone suggest what could be the issue ?
>>>> >>
>>>> >> Thanks
>>>> >> Ajay
>>>> >> --
>>>> >> View this message in context:
>>>> >>
>>>> http://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27755872
>>>> .
>>>> >> html
>>>> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>> >>
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>> >>
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>> >>
>>>> >>
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>>> > For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27771637.html
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27777206.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
