I started porting of test but I have problem with private/protected methods.
Some JLucene methods
are used in tests but marked private in CLucene, e.g.
IndexWriter writer = new IndexWriter(dir, analyzer, true);
writer.addDocument(testDoc);
writer.flush();
SegmentInfo info = writer.newestSegment();
Can be easily ported to
IndexWriter * writer = _CLNEW IndexWriter(dir, analyzer, true);
writer->addDocument(&testDoc);
writer->flush();
SegmentInfo * info = writer->newestSegment();
But the newestSegment method is private, so test cannot be compiled.
Any hint how to go around that?
Borek
> -----Original Message-----
> From: Kostka Bořivoj [mailto:[email protected]]
> Sent: Wednesday, June 23, 2010 5:00 PM
> To: [email protected]
> Subject: Re: [CLucene-dev] vector subscript out of
> rangeexceptionduringindexing
>
> I'll try to port whole TestDocumentsWriter, it is not so big
>
> > -----Original Message-----
> > From: Itamar Syn-Hershko [mailto:[email protected]]
> > Sent: Wednesday, June 23, 2010 12:39 PM
> > To: [email protected]
> > Subject: Re: [CLucene-dev] vector subscript out of range
> > exceptionduringindexing
> >
> > Use Java Lucene 2.3.2, which the git master branch is based on. Grab it from
> > http://archive.apache.org/dist/lucene/java/, or you can use tools like
> > Krugle to read the code on-line.
> >
> > You may only need this to port TestDocumentsWriter as a whole. To fix this
> > specific issue I think it is enough to follow the patch attached to the JIRA
> > issue. I'm not sure it was deployed to the 2.3.2 sources, btw.
> >
> > Itamar.
> >
> > > -----Original Message-----
> > > From: Kostka Bořivoj [mailto:[email protected]]
> > > Sent: Wednesday, June 23, 2010 12:10 PM
> > > To: [email protected]
> > > Subject: Re: [CLucene-dev] vector subscript out of range
> > > exception duringindexing
> > >
> > > I'm not sure which JLucene version I should use (and where to get it)
> > >
> > > Borek
> > >
> > > > -----Original Message-----
> > > > From: Itamar Syn-Hershko [mailto:[email protected]]
> > > > Sent: Wednesday, June 23, 2010 12:11 AM
> > > > To: [email protected]
> > > > Subject: Re: [CLucene-dev] vector subscript out
> > > > ofrangeexceptionduringindexing
> > > >
> > > > Those are the postings array and its staging area for
> > > flushing. Once
> > > > flushed, a Posting object can be deleted.
> > > >
> > > > The code you quoted is originally written in Java as:
> > > > Arrays.fill(postingsFreeList, postingsFreeCount-numToFree,
> > > > postingsFreeCount, null);
> > > >
> > > > Meaning, this is not a deletion but rather a nullification.
> > > This may
> > > > actually be a proper behavior for Java, since it maintains internal
> > > > reference counting of all objects. However, it seem to have caused
> > > > issues with JLucene as well for documents with many terms:
> > > > https://issues.apache.org/jira/browse/LUCENE-1072. Only question is
> > > > how come we haven't seen this until now, and whats special
> > > with the reuters corpus?
> > > >
> > > > I think, if you could port TestDocuemntsWriter to cl_test (at least
> > > > the relevant test case they have added) and check if it
> > > crashes with
> > > > the same characteristics of your issue, we could verify this is the
> > > > same issue. Then we can apply their patch (while following the JIRA
> > > > discussion) accordingly to DocumentsWriter.cpp.
> > > >
> > > > Itamar.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Kostka Bořivoj [mailto:[email protected]]
> > > > > Sent: Tuesday, June 22, 2010 11:53 PM
> > > > > To: [email protected]
> > > > > Subject: Re: [CLucene-dev] vector subscript out of
> > > > > rangeexceptionduringindexing
> > > > >
> > > > > I did some research and found following:
> > > > >
> > > > > The problem is caused by freeing cycle in balanceRAM()
> > > > > (documentswriter.cpp:1325)
> > > > >
> > > > > for ( size_t i =
> > > > > this->postingsFreeCountDW-numToFree;i<
> > > > > this->postingsFreeListDW.length; i++ ){
> > > > > _CLDELETE(this->postingsFreeListDW.values[i]);
> > > > > }
> > > > >
> > > > > Because this->postingsFreeListDW.values contains pointers
> > > which are
> > > > > also used in postingsHash table, the _CLDELETE makes them invalid.
> > > > >
> > > > > So the main question is why Postings objects referenced in
> > > > > postingsHash are also referenced by postingsFreeListDW.
> > > > >
> > > > > Until now I was not able to find the reason.
> > > > >
> > > > > Borek
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Itamar Syn-Hershko [mailto:[email protected]]
> > > > > > Sent: Monday, June 21, 2010 2:08 PM
> > > > > > To: [email protected]
> > > > > > Subject: Re: [CLucene-dev] vector subscript out of range
> > > > > > exceptionduringindexing
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > This seems to be the same error reported by Klemens Friedl
> > > > > last week [1].
> > > > > >
> > > > > > I can confirm your findings. After setting the demo
> > > application to
> > > > > > index the reuters corpora distributed with CLucene (see
> > > my patch
> > > > > > to master today), and setting maxFieldLength to MAX_INT, the
> > > > > applications
> > > > > > is failing on one of the files (for me it was
> > > reut2-002.sgm). Call
> > > > > > stack points to DocumentsWriterThreadState.cpp ln 1142, where
> > > > > > threadState->p is pointing to freed or invalid memory.
> > > > > >
> > > > > > Unfortunately at the moment I cannot work on tracing this
> > > > > properly. If
> > > > > > you can do this yourself, I'll be happy to assist with
> > > > > whatever I can.
> > > > > >
> > > > > > Itamar.
> > > > > >
> > > > > > [1]
> > > > > http://comments.gmane.org/gmane.comp.jakarta.lucene.clucene.de
> > > > vel/3449 .
> > > > > Also see
> > > > >
> > > http://sourceforge.net/tracker/?func=detail&aid=2981449&group_id=800
> > > > > 13
> > > > > &atid=
> > > > > 558446.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Kostka Bořivoj [mailto:[email protected]]
> > > > > > Sent: Monday, June 21, 2010 2:50 PM
> > > > > > To: [email protected]
> > > > > > Subject: [CLucene-dev] vector subscript out of range exception
> > > > > > duringindexing
> > > > > >
> > > > > > During indexing set of documents (about 10000 already
> > > > > > indexed) I get the exception "vector subscript out of range"
> > > > > > from ArrayBase operator [ ].
> > > > > > I did some research and it seems it is because
> > > > > > threadState->postingEquals() method is called with
> > > invalid p set.
> > > > > > The postingsHash[hashPos] probably contains pointer to already
> > > > > > deleted object, as 0xfeee is in all members (I'm
> > > running it under
> > > > > > MSVC 2005 Debugger).
> > > > > > See call stack and threadState->p dump below.
> > > > > >
> > > > > > Source (documentswriterthreadstate.cpp:1010)
> > > > > > ======
> > > > > >
> > > > > > // Locate Posting in hash
> > > > > > threadState->p = postingsHash[hashPos];
> > > > > >
> > > > > > if (threadState->p != NULL &&
> > > > > > !threadState->postingEquals(tokenText, tokenTextLen)) { ...
> > > > > >
> > > > > >
> > > > > > Call stack
> > > > > > ========
> > > > > > clucene-cored.dll!lucene::util::ArrayBase<wchar_t
> > > > > > *>::operator[](unsigned int _Pos=0xfffffbbb) Line 92 C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::ThreadState:
> > > > > > :postingEquals(const wchar_t * tokenText=0x032772a8, const
> > > > > > int tokenTextLen=0x00000008) Line 577 + 0x25 bytes C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::ThreadState:
> > > > > > :FieldData::addPosition(lucene::analysis::Token *
> > > > > > token=0x0100c770) Line 1012 + 0x26 bytes C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::ThreadState:
> > > > > > :FieldData::invertField(lucene::document::Field *
> > > > > > field=0x04d2a9e0, lucene::analysis::Analyzer *
> > > > > > analyzer=0x010a5fa0, const int
> > > > > > maxFieldLength=0x00002710)
> > > > > > Line 902 C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::ThreadState:
> > > > > > :FieldData::processField(lucene::analysis::Analyzer *
> > > > > > analyzer=0x010a5fa0) Line 797 C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::ThreadState:
> > > > > > :processDocument(lucene::analysis::Analyzer *
> > > > > > analyzer=0x010a5fa0) Line 554 + 0x1a bytes C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::updateDocume
> > > > > > nt(lucene::document::Document * doc=0x0012f600,
> > > > > > lucene::analysis::Analyzer * analyzer=0x010a5fa0,
> > > > > > lucene::index::Term * delTerm=0x00000000) Line 934 + 0xc
> > > > > > bytes C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::DocumentsWriter::addDocument(
> > > > > > lucene::document::Document * doc=0x0012f600,
> > > > > > lucene::analysis::Analyzer * analyzer=0x010a5fa0) Line
> > > 919 C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::IndexWriter::addDocument(luce
> > > > > > ne::document::Document * doc=0x0012f600,
> > > > > > lucene::analysis::Analyzer
> > > > > > * analyzer=0x010a5fa0) Line 670 +
> > > > > > 0x13 bytes C++
> > > > > >
> > > > > > clucene-cored.dll!lucene::index::IndexModifier::addDocument(lu
> > > > > > cene::document::Document * doc=0x0012f600,
> > > > > > lucene::analysis::Analyzer * docAnalyzer=0x010a5fa0) Line
> > > > > > 100 C++
> > > > > >
> > > > > > mkidx.exe!tovek::index::Index::indexDocument(tovek::index::Doc
> > > > > > ument & doc={...}, bool bInsert=false, unsigned long &
> > > > > > ulPrevDoc=0x00000007, tovek::analysis::CachedAnalyzer *
> > > > > > pCachedAnalyzer=0x010a5fa0) Line 472 C++
> > > > > >
> > > > > >
> > > > > > Problematic item in PostingHash:
> > > > > > =========================
> > > > > >
> > > > > > - threadState->p 0x02538fd8
> > > > > > {textStart=0xfeeefeee docFreq=0xfeeefeee freqStart=0xfeeefeee
> > > > > > ...} lucene::index::DocumentsWriter::Posting *
> > > > > > textStart 0xfeeefeee int
> > > > > > docFreq 0xfeeefeee int
> > > > > > freqStart 0xfeeefeee int
> > > > > > freqUpto 0xfeeefeee int
> > > > > > proxStart 0xfeeefeee int
> > > > > > proxUpto 0xfeeefeee int
> > > > > > lastDocID 0xfeeefeee int
> > > > > > lastDocCode 0xfeeefeee int
> > > > > > lastPosition 0xfeeefeee int
> > > > > > + vector 0xfeeefeee {p=??? lastOffset=???
> > > > > > offsetStart=??? ...}
> > > > lucene::index::DocumentsWriter::PostingVector *
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > ----------------
> > > > > > ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad
> > > > > > Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky
> > > parental unit.
> > > > > > See the prize list and enter to win:
> > > > > > http://p.sf.net/sfu/thinkgeek-promo
> > > > > > _______________________________________________
> > > > > > CLucene-developers mailing list
> > > > > > [email protected]
> > > > > > https://lists.sourceforge.net/lists/listinfo/clucene-developers
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > --------------------------------------------------------------------
> > > > > --
> > > > > -------- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky
> > > > > parental unit. See the prize list and enter to win:
> > > > > http://p.sf.net/sfu/thinkgeek-promo
> > > > > _______________________________________________
> > > > > CLucene-developers mailing list
> > > > > [email protected]
> > > > > https://lists.sourceforge.net/lists/listinfo/clucene-developers
> > > >
> > > >
> > > ----------------------------------------------------------------------
> > > > ------
> > > > --
> > > > ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad
> > > > Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky
> > > parental unit.
> > > > See the prize list and enter to win:
> > > > http://p.sf.net/sfu/thinkgeek-promo
> > > > _______________________________________________
> > > > CLucene-developers mailing list
> > > > [email protected]
> > > > https://lists.sourceforge.net/lists/listinfo/clucene-developers
> > > >
> > > >
> > > >
> > > ----------------------------------------------------------------------
> > > > -------- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > > lucky parental
> > > > unit. See the prize list and enter to win:
> > > > http://p.sf.net/sfu/thinkgeek-promo
> > > > _______________________________________________
> > > > CLucene-developers mailing list
> > > > [email protected]
> > > > https://lists.sourceforge.net/lists/listinfo/clucene-developers
> > >
> > > --------------------------------------------------------------
> > > ----------------
> > > ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky
> > > parental unit. See the prize list and enter to win:
> > > http://p.sf.net/sfu/thinkgeek-promo
> > > _______________________________________________
> > > CLucene-developers mailing list
> > > [email protected]
> > > https://lists.sourceforge.net/lists/listinfo/clucene-developers
> > >
> >
> >
> > ------------------------------------------------------------------------------
> > ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > lucky parental unit. See the prize list and enter to win:
> > http://p.sf.net/sfu/thinkgeek-promo
> > _______________________________________________
> > CLucene-developers mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit. See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> CLucene-developers mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
CLucene-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/clucene-developers
