[
https://issues.apache.org/jira/browse/LUCENE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416154#comment-13416154
]
Steven Rowe commented on LUCENE-3747:
-------------------------------------
Subversion script for trunk:
{code}
svn rm
lucene/test-framework/src/java/org/apache/lucene/util/TestRuleIcuHack.java
svn rm lucene/analysis/icu/lib/icu4j-4.8.1.1.jar.sha1
svn rm solr/contrib/extraction/lib/icu4j-4.8.1.1.jar.sha1
svn rm solr/contrib/analysis-extras/lib/icu4j-4.8.1.1.jar.sha1
{code}
branch_4x will need more svn moves and version checks for the versioned
grammars.
h2. {{lucene/analysis/common/}}
* I ran {{ant gen-tlds}}
* I ran {{ant jflex}}
h2. {{lucene/analysis/icu/}}
h3. uax29
* I don't fully understand the syntax used in the .rbbi files, so I didn't
check whether they need algorithm updates. (However, since I didn't need to
make any changes for the JFlex version, probably no algorithm changes needed.)
* I ran {{ant genrbbi}}.
h3. utr30
h4. BasicFoldings.txt:
* For those things that are "additions to kd" - how to extend?
* For dashes folding, I added some non-included ranges. Q: should wave dash be
folded to swung dash? (They look the same.)
* I don't know how to extend underline folding - is there a property?
* I don't know how to extend punctuation folding - is there a property?
h4. DiacriticFolding.txt:
* In the [:Diacritic:] section, I'm not sure how to proceed, as I can see
several missing Latin-1 code points that were almost certainly part of Unicode
6.0.0, so the selection mechanism is non-transparent.
* In the [:Mark:]&[:Lm:] section, I'm not sure how to make selections, so I
didn't try.
* In the "Additional Arabic/Hebrew decompositions" section, I don't know how to
extend.
* Other sections were based either on AsciiFoldingFilter or UTR#30, neither of
which has changed
h4. DigbatFolding.txt:
* based on AsciiFoldingFilter, which hasn't changed
h4. HanRadicalFolding.txt:
* based on UTR#30, which hasn't changed
h4. NativeDigitFolding.txt:
* I wrote a shell/perl script, embedded in the text file, to update.
* Should [:No:] DIGIT chars be included? One currently is: {{19DA;NEW TAI LUE
THAM DIGIT ONE;No;0;L;;;1;1;N;;;;;}}, but others are not (other ranges listed
in the patch).
h3. nfkc.txt:
* New version copied directly from icu-project.org
* There is a problem: the following from {{TestICUFoldingFilter}} fails:
{code:java}
46: assertAnalyzesTo(a, "Μάϊος", new String[] { "μαιοσ" });
{code}
AFAICT, this is because the accent decomposition mappings are no longer
present in nfkc.txt. This makes no sense; Robert, do you know what's happening
here?
h3. nfkc_cf.txt:
* New version copied directly from icu-project.org
> Support Unicode 6.1.0
> ---------------------
>
> Key: LUCENE-3747
> URL: https://issues.apache.org/jira/browse/LUCENE-3747
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.5, 4.0-ALPHA
> Reporter: Steven Rowe
> Priority: Minor
> Attachments: LUCENE-3747.patch
>
>
> Now that Unicode 6.1.0 has been released, Lucene/Solr should support it.
> JFlex trunk now supports Unicode 6.1.0.
> Tasks include:
> * Upgrade ICU4J to v49 (after it's released, on 2012-03-21, according to
> http://icu-project.org).
> * Use {{icu}} module tools to regenerate the supplementary character
> additions to JFlex grammars.
> * Version the JFlex grammars: copy the current implementations to
> {{*Impl3<X>}}; cause the versioning tokenizer wrappers to instantiate this
> version when the {{Version}} c-tor param is in the range 3.1 to the version
> in which these changes are released (excluding the range endpoints); then
> change the specified Unicode version in the non-versioned JFlex grammars from
> 6.0 to 6.1.
> * Regenerate JFlex scanners, including {{StandardTokenizerImpl}},
> {{UAX29URLEmailTokenizerImpl}}, and {{HTMLStripCharFilter}}.
> * Using {{generateJavaUnicodeWordBreakTest.pl}}, generate and then run
> {{WordBreakTestUnicode_6_1_0.java}} under
> {{modules/analysis/common/src/test/org/apache/lucene/analysis/core/}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
