[
https://issues.apache.org/jira/browse/LUCENE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416196#comment-13416196
]
Robert Muir commented on LUCENE-3747:
-------------------------------------
{quote}
For those things that are "additions to kd" - how to extend?
{quote}
What this means is that its the mappings in utr30 minus what nfkc does.
So i dont think we need to really worry about this much?
if we wanted we could check the sets present for some, defined by the below
link:
http://www.unicode.org/reports/tr30/tr30-4.html#_Toc42
So thats sorta the rule for this whole file?
Example:
{noformat}
## Space Folding
# [:Zs:] > U+0020
1680>0020
180E>0020
{noformat}
So that is basically iterating this UnicodeSet (can be done in code with ICU)
and generating mappings to 0020:
{noformat}
[[:Zs:]-[\u0020]-[:Changes_When_NFKC_Casefolded=Yes:]]
{noformat}
Does that make sense?
{quote}
Should [:No:] DIGIT chars be included? One currently is: 19DA;NEW TAI LUE THAM
DIGIT ONE;No;0;L;;;1;1;N;;;;;, but others are not (other ranges listed in the
patch).
{quote}
I dont see any problems including [:Numeric_Type=Digit:]. But i wouldnt use
[:No:].
So something like [[:Numeric_Type=Digit:][:Nd:]] ?
{quote}
In the [:Diacritic:] section, I'm not sure how to proceed, as I can see several
missing Latin-1 code points that were almost certainly part of Unicode 6.0.0,
so the selection mechanism is non-transparent.
{quote}
Its definitely a subset. Some stuff in here like viramas should not be folded
away.
Sorry i dont have a well-defined set or criteria, it was just common-sense.
For the last update, i basically only reviewed the 'new' ones and made a
decision: e.g.
{noformat}
[[:Diacritic:]-[:Age=6.0:]]
{noformat}
So this will be the trickiest part of the file to automate i think, as it was
originally
defined as a list for the most part:
http://www.unicode.org/reports/tr30/datafiles/DiacriticFolding.txt
> Support Unicode 6.1.0
> ---------------------
>
> Key: LUCENE-3747
> URL: https://issues.apache.org/jira/browse/LUCENE-3747
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.5, 4.0-ALPHA
> Reporter: Steven Rowe
> Priority: Minor
> Attachments: LUCENE-3747.patch
>
>
> Now that Unicode 6.1.0 has been released, Lucene/Solr should support it.
> JFlex trunk now supports Unicode 6.1.0.
> Tasks include:
> * Upgrade ICU4J to v49 (after it's released, on 2012-03-21, according to
> http://icu-project.org).
> * Use {{icu}} module tools to regenerate the supplementary character
> additions to JFlex grammars.
> * Version the JFlex grammars: copy the current implementations to
> {{*Impl3<X>}}; cause the versioning tokenizer wrappers to instantiate this
> version when the {{Version}} c-tor param is in the range 3.1 to the version
> in which these changes are released (excluding the range endpoints); then
> change the specified Unicode version in the non-versioned JFlex grammars from
> 6.0 to 6.1.
> * Regenerate JFlex scanners, including {{StandardTokenizerImpl}},
> {{UAX29URLEmailTokenizerImpl}}, and {{HTMLStripCharFilter}}.
> * Using {{generateJavaUnicodeWordBreakTest.pl}}, generate and then run
> {{WordBreakTestUnicode_6_1_0.java}} under
> {{modules/analysis/common/src/test/org/apache/lucene/analysis/core/}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
