[
https://issues.apache.org/jira/browse/LUCENE-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615775#action_12615775
]
Steven Rowe commented on LUCENE-1343:
-------------------------------------
Hi Robert,
My comments below assume you're intrestested in having this code hosted in the
Lucene source repository - please disregard if that's not the case.
Have you seen the [HowToContribute page on the Lucene
wiki|http://wiki.apache.org/lucene-java/HowToContribute]? It outlines some of
the basics concerning code submissions.
A couple of things I noticed that need to be addressed before the code will be
accepted:
# Tab characters should be converted to spaces
# Indentation increment should be two spaces
# Test(s) should be moved from the UnicodeNormalizationFilterFactory.main()
method into standalone class(es) that extend LuceneTestCase
# More/more explicit javadocs - for example, you should describe the set of
provided transformations (e.g. Cyrillic diacritic stripping is included).
# Solr is a separate code base, so the UnicodeNormalizationFilterFactory should
be moved to a Solr JIRA issue
# Because it has a dependency on the ICU jar, this contribution will have to
live in the contrib/ area -- the Java packages name should be adjusted
accordingly.
# The submission should be repackaged as a patch (instructions available on the
above-linked wiki page).
> A replacement for ISOLatin1AccentFilter that does a more thorough job of
> removing diacritical marks or non-spacing modifiers.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1343
> URL: https://issues.apache.org/jira/browse/LUCENE-1343
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Robert Haschart
> Priority: Minor
> Attachments: normalizer.jar, UnicodeCharUtil.java,
> UnicodeNormalizationFilter.java, UnicodeNormalizationFilterFactory.java
>
>
> The ISOLatin1AccentFilter takes Unicode characters that have diacritical
> marks and replaces them with a version of that character with the diacritical
> mark removed. For example é becomes e. However another equally valid way of
> representing an accented character in Unicode is to have the unaccented
> character followed by a non-spacing modifier character (like this: é )
> The ISOLatin1AccentFilter doesn't handle the accents in decomposed unicode
> characters at all. Additionally there are some instances where a word will
> contain what looks like an accented character, that is actually considered to
> be a separate unaccented character such as Ł but which to make searching
> easier you want to fold onto the latin1 lookalike version L .
> The UnicodeNormalizationFilter can filter out accents and diacritical marks
> whether they occur as composed characters or decomposed characters, it can
> also handle cases where as described above characters that look like they
> have diacritics (but don't) are to be folded onto the letter that they look
> like ( Ł -> L )
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
