I've been working off the head of CLucene (which has great!) and I ran into a
memory smasher.
I had strange issues where after some queries the search results would start to
get more and more incorrect until the app would crash. After much debugging I
was able to confirm that this would only occur for queries which had lengths
that were multiples of 8.
After even more debugging I found that the KeywordTokenizer (which I was using
for my queries) was allocating term buffers that where multiples of 8
(suspicious!). It turns out that if the token length is a multiple of 8 and
the KeywordTokenizer attempts to null terminate the string, it writes off the
end of the array, causing memory corruption. Normally you don't see this
because it silently corrupts and the token must be the multiple of 8.
To fix this I make sure to add room for the null terminator if the buffer is
already full. Here is my patch:
diff --git a/src/core/CLucene/analysis/Analyzers.cpp
b/src/core/CLucene/analysis/Analyzers.cpp
index 0c34a60..39fec43 100644
--- a/src/core/CLucene/analysis/Analyzers.cpp
+++ b/src/core/CLucene/analysis/Analyzers.cpp
@@ -556,6 +556,9 @@ Token* KeywordTokenizer::next(Token* token){
if ( termBuffer == NULL ){
termBuffer=token->resizeTermBuffer(token->bufferLength() + 8);
}
+ if (upto == token->bufferLength()) {
+ termBuffer = token->resizeTermBuffer(token->bufferLength() + 1);
+ }
termBuffer[upto]=0;
token->setTermLength(upto);
return token;
Let me know if I should open a bug for this.
Thanks again for clucene!
--
Matt Ronge
Central Atomics
Makers of Rocketbox
http://www.getrocketbox.com
mro...@centralatomics.com
------------------------------------------------------------------------------
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
_______________________________________________
CLucene-developers mailing list
CLucene-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clucene-developers
