DM Smith wrote:
According to the documentation for TokenStream, derived classes are
to override either next() or next(Token).
Currently, if next(Token) is overridden, but next() is called,
payload is cloned if it exists in the new token.
However, if next(Token) is called, it is up to the implementation to
properly maintain payload. It seems to me that since next(Token) is
called with a new Token(), that this should not be necessary.
I see: you mean that extra clone of the payload that we do in
TokenStream.next() should not be necessary? We added it for some
reason -- oh it was done as part of LUCENE-1062. I think the
challenge is we can't be sure the TokenStream isn't sharing a single
payload, even though we passed in a new Token, because the reuse API
allows you to reuse Token and payload. So it's safer to clone it
because next() is supposed to provide a fully unshared Token.
Also, if next() is overridden, but next(Token) is called, then
payload is never cloned, unless the overriding method handles it.
Well, in this case, if you override next(), you are supposed to return
a fully unshared Token, so the payload should already be "private"? I
think?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
