Alrighty, seems like I have nailed it. See below + attached patch.
On 29/6/2010 12:39 AM, Kostka Bořivoj wrote:
I'm quite sure the problem is in postingsFreeListDW management:
The postings after postingsFreeCountDW are used somewhere (but are still here
in a list). If you
remove block of free postings in balanceRAM, you are removing before
postingsFreeCountDW
so an "empty block" is created inside postingsFreeListDW (at the end of free
postings but before
all used postings). Then if you return all/some used postings back into list,
you return them to the beginning
of this empty block. This can cause duplicated pointers in the list.
I agree there is an empty block, I am just not sure how this could cause
duplicated pointers?
Also, claiming the culprit is postingsFreeListDW's memory management is
less likely since it most likely would have been noticed by Java users /
devs already.
It can also be a reason, why postingsFreeCountDW< postingsFreeListDW.length in
the destructor.
I am not at all sure this is an actual problem. postingsFreeListDW can
have more allocated memory than needed (hence < length).
Borek
Anyway, I put a breakpoint on Array.h ln 139 and noticed it errored on
array item 5888 (0-base). This was also the value of
postingsFreeCountDW. What this means is the "gap" you were referring to
is actually in the tails of the postings array. Some code moving
postings around didn't bother at NULLifying unused positions.
Instead of looking up this erroneous code, I added a memset call to
NULLify the tail of that array at the end of the destructor. It also
makes more sense performance-wise, and as long as no memory leaks are
introduced, I think it is safe to keep it that way.
So, what we need now is to make sure this doesn't introduce new leaks,
and have this properly tested before checking this change in (all
cl_tests pass atm). I know cl_demo leaks but haven't had the time to see
where from - this could be related. Can you handle that?
Itamar.
commit 7413f49ccdf24330fc6e06efaa4c5fa632f3fb37
Author: synhershko <synhers...@users.sourceforge.net>
Date: Tue Jun 29 01:28:24 2010 +0300
Fixing a crash on DocumentsWriter dtor when writer->setRAMBufferSizeMB is
used
diff --git a/src/core/CLucene/index/DocumentsWriter.cpp
b/src/core/CLucene/index/DocumentsWriter.cpp
index c5c6e32..45ab3d6 100644
--- a/src/core/CLucene/index/DocumentsWriter.cpp
+++ b/src/core/CLucene/index/DocumentsWriter.cpp
@@ -118,6 +118,11 @@ DocumentsWriter::~DocumentsWriter(){
for(size_t i=0;i<threadStates.length;i++) {
_CLDELETE(threadStates.values[i]);
}
+
+ // Make sure unused posting slots aren't attempted delete on
+ memset(this->postingsFreeListDW.values + this->postingsFreeCountDW
+ , NULL
+ , (this->postingsFreeListDW.length - this->postingsFreeCountDW) *
sizeof(Posting*));
}
void DocumentsWriter::setInfoStream(std::ostream* infoStream) {
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
CLucene-developers mailing list
CLucene-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clucene-developers
