"Yonik Seeley" <[EMAIL PROTECTED]> wrote: > On Nov 19, 2007 2:03 PM, Michael McCandless <[EMAIL PROTECTED]> > wrote: > > Maybe you mean that each Token must be fully independent because there > > are plenty of filters that hold onto each Token long after next() is > > called again, and then serve them up again later > > Yes.
OK. > > But this is why we have the Token next() method (which returns a > > "stable" Token that the callee can't change later) vs the Token > > next(Token) method which allows the callee to re-use the passed in > > Token on the next call. > > Right... that works great for termBuffer, but the difference is that > Token owns the termBuffer (contents are always copied into it). Right. > > So eg in TokenStream.next() base implementation we would have to make > > a new copy of the payload byte[] and set it in the copied token. This > > way when a caller calls next() they get a full private copy, even of > > the by-reference payload. > > Perhaps I'm the one missing something, but it still doesn't seem like > this works a filter can set the reference to the byte[] and is allowed > to reuse it. The particular filter setting the payload is in control, > not next(). If the filter implements next() then that means it will never change the byte[] payload that it set on the token -- the "private copy" API. But if it implements next(Token) then it is allowed to change it on the next call to next(Token) -- the "allow reuse" API. So if a newish filter implements the "reuse" API, but an older filter comes after it in the chain and calls next(), that older filter gets a full copy because the default next() in TokenStream makes the copy. I think? Mike --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
