[
https://issues.apache.org/jira/browse/LUCENE-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759568#action_12759568
]
Uwe Schindler commented on LUCENE-1926:
---------------------------------------
bq. btw, my TestCase uses StandardTokenizer, which does call clearAttributes().
I have seen this, because of that the attrs between step 0 and 1 are cleared.
As you do not call incrementToken in the underlying filter in step 2, it seems
to be preserved (in fact, you are the source of tokens and should call
clearAttributes() for this step).
The problem with preserving the attribute state between calls to incrementToken
is e.g. the following even with incrementToken():
Just put an ReverseTokenFilter on top of this TokenFilter. This tokenfilter
reverses the term. If you only consume with incrementToken() and rely on the
fact that the tokens from the last call are preserved, you fail: The Token is
reversed by the reverse filter and then step 2 would then see the reversed term
text and not the forward one exspected from step 1.
If you want to preserve states between incrementToken calls, you have to
capture the state. Maybe the Javadocs should be extended, to clearly note, that
attribute contents (may) not preserved between calls to incrementToken().
> Back compat break with old next() consumer API
> ----------------------------------------------
>
> Key: LUCENE-1926
> URL: https://issues.apache.org/jira/browse/LUCENE-1926
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.9
> Reporter: Robert Muir
> Attachments: CaptureStateTestcase.java
>
>
> There is a bug that causes tokenstreams to return different results,
> depending upon whether they are consumed with the incrementToken() api or the
> next() api.
> I found this because the Solr analysis tool in the admin page uses the next()
> api, and i was seeing strange results.
> I've created a test case to show the problem. when calling captureState(),
> the current state is erased, but only when consuming with the next() api.
> If I consume with incrementToken(), things work.
> {code}
> State tempState = captureState(); // after we capture state here, things get
> strange.
> String right = termAtt.term(); // when using old consumer API, this value is
> wrong!!!!
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
