[
https://issues.apache.org/jira/browse/LUCENE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761254#comment-13761254
]
Benson Margulies commented on LUCENE-5202:
------------------------------------------
Yes, that's what I have and it works, except for the problem I wrote this test
case to demonstrate. There's a call to peekToken in nextToken used to detect
the end of the input. When that gets called, a token 'moves' from the input to
the positions, so the calls to peekToken in my code never see it.
Either I'm supposed to call restoreState to examine it, or there's a problem
here. If I'm supposed to call restoreState, I need to figure out how to notice
(by looking at positions?) that I'm in that situation. Or there's some problem
in my logic for deciding when to do my next load of peeks, so that nextToken is
never supposed to reach that call to peek, but I can't figure out what it is.
> LookaheadTokenFilter consumes an extra token in nextToken
> ---------------------------------------------------------
>
> Key: LUCENE-5202
> URL: https://issues.apache.org/jira/browse/LUCENE-5202
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 4.3.1
> Reporter: Benson Margulies
> Attachments: LUCENE-5202.patch, LUCENE-5202.patch
>
>
> This is a bit hard to explain except by looking at the test case. I've coded
> a filter that uses LookaheadTokenFilter. The incrementToken method peeks some
> tokens. Then, it seems, nextToken in the Lookahead class calls peekToken
> itself, which seems to me to consume a token so that it's not seen when the
> derived class sets out to process the next set of tokens.
> In passing, this test case can be used to demonstrate that it does not work
> to try to use the afterPosition method to set up attributes of the token that
> we're 'after'. Probably that was never intended. However, I'm hoping for some
> feedback as to whether the rest of the structure here is as intended for
> subclasses of LookaheadTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
