[
https://issues.apache.org/jira/browse/LUCENE-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905758#comment-13905758
]
ASF subversion and git services commented on LUCENE-5447:
---------------------------------------------------------
Commit 1569855 from [~steve_rowe] in branch 'dev/branches/lucene_solr_4_7'
[ https://svn.apache.org/r1569855 ]
LUCENE-5447: Fully patch StandardTokenizerImpl.jflex, to bring parity with
rules in UAX29URLEmailTokenizerImpl.jflex; add tests that fail without this fix
and succeed with it. (merged branch_4x r1569849)
> StandardTokenizer should break at consecutive chars matching Word_Break =
> MidLetter, MidNum and/or MidNumLet
> ------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-5447
> URL: https://issues.apache.org/jira/browse/LUCENE-5447
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 4.6.1
> Reporter: Steve Rowe
> Assignee: Steve Rowe
> Fix For: 4.7, 5.0
>
> Attachments: LUCENE-5447-take2.patch, LUCENE-5447-test.patch,
> LUCENE-5447.patch, LUCENE-5447.patch
>
>
> StandardTokenizer should split all of the following sequences into two tokens
> each, but they are all instead kept intact and output as single tokens:
> {noformat}
> "A::B" (':' is in \p{Word_Break = MidLetter})
> "1..2", "A..B" ('.' is in \p{Word_Break = MidNumLet})
> "A.:B"
> "A:.B"
> "1,,2" (',' is in \p{Word_Break = MidNum})
> "1,.2"
> "1.,2"
> {noformat}
> Unfortunately, the word break test data released with Unicode, e.g. for
> Unicode 6.3
> [http://www.unicode.org/Public/6.3.0/ucd/auxiliary/WordBreakTest.txt], and
> incorporated into a versioned Lucene test, e.g.
> {{WordBreakTestUnicode_6_3_0}}, doesn't cover these cases.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
