[
https://issues.apache.org/jira/browse/LUCENE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13767238#comment-13767238
]
Hayden Muhl commented on LUCENE-5211:
-------------------------------------
Ah, very good. I was a bit shocked when my French stop words weren't working.
This seemed like too big of a functionality bug to be easily missed.
> StopFilterFactory docs do not advertise/explain hte "format" option
> -------------------------------------------------------------------
>
> Key: LUCENE-5211
> URL: https://issues.apache.org/jira/browse/LUCENE-5211
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 4.2
> Reporter: Hayden Muhl
> Assignee: Hoss Man
> Priority: Minor
>
> StopFilterFactory supports a "format" option for controlling wether
> "getWordSet" or "getSnowballWordSet" is used to parse the file, but this
> option is not advertised and people can be confused by looking at the example
> stopword files include in the releases (some of which are in the snoball
> format w/ "|" comments) and try to use them w/o explicitly specifying
> {{format="snowball"}} and silently get useless stopwords (that include the "|
> comments" as literal portions of hte stopwrds.
> we need to better document the use of "format" and consider updating all of
> the example stopword files we ship that are in the snowball format with a
> note about the need to use {{format="snowball"}} with those files.
> {panel:title=Initial Bug Report}
> The StopFilterFactory builds a CharArraySet directly from the raw lines of
> the supplied words file. This causes a problem when using the stop word files
> supplied with the Solr/Lucene distribution. In particular, the comments in
> those files get added to the CharArraySet. A line like this...
> ceci | this
> Should result in the string "ceci" being added to the CharArraySet, but "ceci
> | this" is what actually gets added.
> Workaround: Remove all comments from stop word files you are using.
> Suggested fix: The StopFilterFactory should strip any comments, then strip
> trailing whitespace. The stop word files supplied with the distribution
> should be edited to conform to the supported comment format.
> {panel}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
