Thanks. I also suppose there is no such problem in the code, but I didn't want to throw this away without discussing with somebody from CLucene team.
I'll continue testing cl_demo and try to find where are the leaks come from. and also check what's wrong with testWickedLongTerm(). Borek > -----Original Message----- > From: Itamar Syn-Hershko [mailto:ita...@code972.com] > Sent: Tuesday, June 29, 2010 1:57 PM > To: clucene-developers@lists.sourceforge.net > Subject: Re: [CLucene-dev] vector subscript outofrangeexceptionduringindexing > > On 29/6/2010 1:41 PM, Kostka Bořivoj wrote: > > I agree it works fine (and your way of nulling is definitelly better than > > mine). > > I already indexed about 1GB of data, but I'm not sure about mem leaks, > > as my application memory increases constantly during indexing (and it > > didn't with > previous > > version). But this could be of course bug in my app, or just becase the > > size of > internal > > "caches" increases or something else... > > > This is why I said using cl_demo and cl_test as a testbed to look for > crashes and memory leaks is better. > > This also depends on how you measure the memory usage for your > application. Windows' Task manager isn't reliable for that. > > > Before you mar thi solved, I'will try to explain, what I meant by duplicate > > pointers etc. > In one of my previous mails, > > just for case you find this important. If this is problem or not strongly > > depends on how > postingsFreeListDW should be used. > > If only items from 0 to postingsFreeCountDW are considered valid and the > > rest is > just "junk" everything is OK. > > But if postingsFreeListDW is supposed to be used beyond the > postingsFreeCountDW (eg. for freeing postings objects during destruction), we > can > run into the problems. > > > > ... > > > > The question is if > > > > (1) the tail always contains junk during destruction (and everything is OK, > > if yes) or > > > > (2) it can also contain valid pointers which should be deleted (so memory > > leaks are > produced by tail nulling) > > > > Unfortunatelly until now I was not able figure out if (1) or (2) is true. > > > To correctly answer that, I need to profile the code which is something > I cannot do at the moment. I am not the original author of the code - > neither in its Java nor C++ shape. If you want to do this, use > infoStream for easier tracking. > > However, I think the way it operates is designed not to have such a > scenario. Although Java won't suffer from double-deletions, that class > was designed not to hold unused objects when they aren't needed - which > is essentially what you are describing here. > > The answer to your questions will be in the actual place where copying > is performed. See DocumentsWriter::getPostings. I think the way copy > operations are performed there prevents this from happening. Fact is on > destruction all valid objects are in the head of the array, and the tail > contains nothing important. That is at least from the use cases in > cl_demo and cl_test. > > The best way to either confirm or rule this is by thoroughly testing > this - while both looking for crashes and mem-leaks in various scenarios > (multi-threaded?). > > If you wish, you could post such a question on Java Lucene's dev list. > They're sure to answer this correctly, if they'll remember what they did > there back in '07 or so :) > > HTH > > Itamar > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > CLucene-developers mailing list > CLucene-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/clucene-developers ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first _______________________________________________ CLucene-developers mailing list CLucene-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/clucene-developers
