[
https://issues.apache.org/jira/browse/LUCENE-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715786#action_12715786
]
Robert Muir commented on LUCENE-1377:
-------------------------------------
i don't really know enough to say that, but this one is especially important
(imho)
without getting into details, the changes Yonik made to WordDelimiterFilter, in
combination with WhitespaceTokenizer treat the various unicode categories
correctly, unlike any of the other analyzers in lucene.
No, it doesn't actually use the Word_Break properties which is really the key,
but if you toss some Hindi, Bangla, Tibetan, Arabic, Tamil, ... [list goes on
very long] text at it, in general it's gonna work pretty well. This is a big
improvement over any of the other default options!
> Add HTMLStripReader and WordDelimiterFilter from SOLR
> -----------------------------------------------------
>
> Key: LUCENE-1377
> URL: https://issues.apache.org/jira/browse/LUCENE-1377
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 2.3.2
> Reporter: Jason Rutherglen
> Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> SOLR has two classes HTMLStripReader and WordDelimiterFilter which are very
> useful for a wide variety of use cases. It would be good to place them into
> core Lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
