[
https://issues.apache.org/jira/browse/LUCENE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988621#action_12988621
]
Simon Willnauer commented on LUCENE-2901:
-----------------------------------------
bq. If a token is already marked as a keyword (KeywordAttribute.isKeyword() ==
true), perhaps the KeywordMarkerFilterFactory should not re-set the state to
false.
hey Drew, I think I see you point here but I disagree that we should consider
the previous state of the KeywordAttribute in this filter. IMO this filter is
very clearly defined and extremely simple. If you want it to have different
behavior eg. considering the attrs state you can simply write your own filter
that. This one is more or less just a simple PoC how the KeywordAttribute works
/ should be used etc.
I think if we change the behavior we will have JIRA issues that say
"KeywordMarkerFilter does not reset attribute state to false for tokens not in
protwords.txt" - you know what I mean, this one is tricky and special
use-cases should have their own kind of trivial impls.
> KeywordMarkerFilter resets keyword attribute state to false for tokens not in
> protwords.txt
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-2901
> URL: https://issues.apache.org/jira/browse/LUCENE-2901
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 3.1
> Reporter: Drew Farris
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2901.patch
>
>
> KeywordMarkerFilter sets true or false for the KeywordAttribute on all
> tokens. This erases previous state established further up the filter chain,
> for example in the case where a custom filter wants to prevent a token from
> being stemmed.
> If a token is already marked as a keyword (KeywordAttribute.isKeyword() ==
> true), perhaps the KeywordMarkerFilterFactory should not re-set the state to
> false.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
