This is now merged to master. Thanks for reporting!
Itamar. On 5/12/2010 7:43 PM, Matt Ronge wrote: > Shoot I wish I had noticed this earlier: > http://clucene.git.sourceforge.net/git/gitweb.cgi?p=clucene/clucene;a=commit;h=de5695332badddc264c3e187350463d9d6ee4a8a > > Looks like someone else had already found and fixed the bug. I wish I had > found that, would have saved me a lot of time, oh well. > > On Dec 4, 2010, at 1:40 PM, Matt Ronge wrote: > >> I've been working off the head of CLucene (which has great!) and I ran into >> a memory smasher. >> >> I had strange issues where after some queries the search results would start >> to get more and more incorrect until the app would crash. After much >> debugging I was able to confirm that this would only occur for queries which >> had lengths that were multiples of 8. >> >> After even more debugging I found that the KeywordTokenizer (which I was >> using for my queries) was allocating term buffers that where multiples of 8 >> (suspicious!). It turns out that if the token length is a multiple of 8 and >> the KeywordTokenizer attempts to null terminate the string, it writes off >> the end of the array, causing memory corruption. Normally you don't see this >> because it silently corrupts and the token must be the multiple of 8. >> >> To fix this I make sure to add room for the null terminator if the buffer is >> already full. Here is my patch: >> >> diff --git a/src/core/CLucene/analysis/Analyzers.cpp >> b/src/core/CLucene/analysis/Analyzers.cpp >> index 0c34a60..39fec43 100644 >> --- a/src/core/CLucene/analysis/Analyzers.cpp >> +++ b/src/core/CLucene/analysis/Analyzers.cpp >> @@ -556,6 +556,9 @@ Token* KeywordTokenizer::next(Token* token){ >> if ( termBuffer == NULL ){ >> termBuffer=token->resizeTermBuffer(token->bufferLength() + 8); >> } >> + if (upto == token->bufferLength()) { >> + termBuffer = token->resizeTermBuffer(token->bufferLength() + 1); >> + } >> termBuffer[upto]=0; >> token->setTermLength(upto); >> return token; >> >> >> Let me know if I should open a bug for this. >> >> Thanks again for clucene! >> -- >> Matt Ronge >> Central Atomics >> Makers of Rocketbox >> >> http://www.getrocketbox.com >> mro...@centralatomics.com >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> What happens now with your Lotus Notes apps - do you make another costly >> upgrade, or settle for being marooned without product support? Time to move >> off Lotus Notes and onto the cloud with Force.com, apps are easier to build, >> use, and manage than apps on traditional platforms. Sign up for the Lotus >> Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d >> _______________________________________________ >> CLucene-developers mailing list >> CLucene-developers@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/clucene-developers > -- > Matt Ronge > Central Atomics > Makers of Rocketbox > > http://www.getrocketbox.com > mro...@centralatomics.com > > > > > > > > ------------------------------------------------------------------------------ > What happens now with your Lotus Notes apps - do you make another costly > upgrade, or settle for being marooned without product support? Time to move > off Lotus Notes and onto the cloud with Force.com, apps are easier to build, > use, and manage than apps on traditional platforms. Sign up for the Lotus > Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d > _______________________________________________ > CLucene-developers mailing list > CLucene-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/clucene-developers > > ------------------------------------------------------------------------------ Learn how Oracle Real Application Clusters (RAC) One Node allows customers to consolidate database storage, standardize their database environment, and, should the need arise, upgrade to a full multi-node Oracle RAC database without downtime or disruption http://p.sf.net/sfu/oracle-sfdevnl _______________________________________________ CLucene-developers mailing list CLucene-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/clucene-developers
