[
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452676#comment-16452676
]
Alan Woodward commented on LUCENE-8273:
---------------------------------------
bq. You mean any filter that uses captureState?
capture/restoreState works fine, the problem comes when you get a filter that
needs to look ahead in the tokenstream, so for example if SynonymGraphFilter
has a multiword synonym "a b c -> d", and you hit token "a", then the filter
pulls in two more tokens to see if it matches the whole synonym; but
ConditionalTokenFilter only allows you to pull in one token at a time, because
it needs to distinguish between incrementToken() as called by the next filter
down the line, and incrementToken() as called by its delegate.
> Add a ConditionalTokenFilter
> ----------------------------
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
> Issue Type: New Feature
> Reporter: Alan Woodward
> Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265. It would be useful to be able to wrap a TokenFilter
> in such a way that it could optionally be bypassed based on the current state
> of the TokenStream. This could be used to, for example, only apply
> WordDelimiterFilter to terms that contain hyphens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
