[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929036#comment-16929036 ] ASF subversion and git services commented on LUCENE-8966: - Commit ec1ef2bce652686416a6b1d9f57da7805fb1f93c in lucene-solr's branch refs/heads/master from jimczi [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ec1ef2b ] LUCENE-8966: update CHANGES.txt after backport > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929035#comment-16929035 ] ASF subversion and git services commented on LUCENE-8966: - Commit c4815f04c06256dbc9b28afdeb8b9c689198fd7d in lucene-solr's branch refs/heads/branch_8x from jimczi [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c4815f0 ] LUCENE-8966: The Korean analyzer now splits tokens on boundaries between digits and alphabetic characters. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929032#comment-16929032 ] ASF subversion and git services commented on LUCENE-8966: - Commit c8f36238ab0d99e2dc60b944030f90832b850737 in lucene-solr's branch refs/heads/master from jimczi [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c8f3623 ] LUCENE-8966: The Korean analyzer split tokens on boundaries between digits and alphabetic characters. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927933#comment-16927933 ] Namgyu Kim commented on LUCENE-8966: Oh, Thank you for your reply. [~jim.ferenczi] :D I checked again and it was not bug. That result is come from viterbi path. But I think it needs to be discussed. So I added a new issue about it. I'd appreciate if you check LUCENE-8977. P.S. +1 to your patch > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[
https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925588#comment-16925588
]
Jim Ferenczi commented on LUCENE-8966:
--
I don't think it's a bug [~danmuzi] or at least that it's related to this
issue. In your example the first dot ('.' is a word dictionary) is considered a
better path than grouping all dots eagerly. We process the unknown words
greedily so we compare the path "[4], [.], [.]" with "[4], [.], [.],
[]", "[4], [.], [.], [.], [...]", ... "[4], [..]". Keeping the first
dot separated from the rest indicates that a number followed by a dot is a
better splitting path than multiple dots in our model. We can discuss this
behavior in a new issue if you think this should be configurable (for instance
the JapaneseTokenizer process unknown words greedily only in search mode) ?
> KoreanTokenizer should split unknown words on digits
>
>
> Key: LUCENE-8966
> URL: https://issues.apache.org/jira/browse/LUCENE-8966
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8966.patch, LUCENE-8966.patch
>
>
> Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer
> groups characters of unknown words if they belong to the same script or an
> inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the
> rest in Latin) but this rule doesn't work well on digits since they are
> considered common with other scripts. For instance the input "44사이즈" is kept
> as is even though "사이즈" is part of the dictionary. We should restore the
> original behavior and splits any unknown words if a digit is followed by
> another type.
> This issue was first discovered in
> [https://github.com/elastic/elasticsearch/issues/46365]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[
https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924773#comment-16924773
]
Namgyu Kim commented on LUCENE-8966:
But there is a bug I just checked :(
Input : "4..4사이즈"
Expected : [4] [..] [4] [사이즈]
Actual : [4] *[.] [.]* [4] [사이즈]
{code:java}
// Need to pass!
public void testDuplicatePunctuation() throws IOException {
assertAnalyzesTo(analyzerWithPunctuation, "4..4사이즈",
new String[]{"4", "..", "4", "사이즈"},
new int[]{0, 1, 7, 8},
new int[]{1, 7, 8, 11},
new int[]{1, 1, 1, 1}
);
}
{code}
I think we need to fix it.
If it is okay to fix within this JIRA issue, I'll post additional patch.
Otherwise I'll create a new one.
> KoreanTokenizer should split unknown words on digits
>
>
> Key: LUCENE-8966
> URL: https://issues.apache.org/jira/browse/LUCENE-8966
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8966.patch, LUCENE-8966.patch
>
>
> Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer
> groups characters of unknown words if they belong to the same script or an
> inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the
> rest in Latin) but this rule doesn't work well on digits since they are
> considered common with other scripts. For instance the input "44사이즈" is kept
> as is even though "사이즈" is part of the dictionary. We should restore the
> original behavior and splits any unknown words if a digit is followed by
> another type.
> This issue was first discovered in
> [https://github.com/elastic/elasticsearch/issues/46365]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924768#comment-16924768 ] Namgyu Kim commented on LUCENE-8966: Good job! [~jim.ferenczi] :D It can be serious enough for Nori users. About Punctuation, as [~jim.ferenczi] said, it can be remained by using discardPunctuation(set false) parameter in KoreanTokenizer. You can test it by using analyzerWithPunctuation instance in TestKoreanTokenizer. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924206#comment-16924206 ] Mike Sokolov commented on LUCENE-8966: -- > For complex number grouping and normalization, Namgyu Kim added a > KoreanNumberFilter in https://issues.apache.org/jira/browse/LUCENE-8812 Ah thanks, I'll have a look > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[
https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923404#comment-16923404
]
Lucene/Solr QA commented on LUCENE-8966:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} nori in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-8966 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979562/LUCENE-8966.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 78b6530fb26 |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/206/testReport/ |
| modules | C: lucene/analysis/nori U: lucene/analysis/nori |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/206/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> KoreanTokenizer should split unknown words on digits
>
>
> Key: LUCENE-8966
> URL: https://issues.apache.org/jira/browse/LUCENE-8966
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8966.patch, LUCENE-8966.patch
>
>
> Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer
> groups characters of unknown words if they belong to the same script or an
> inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the
> rest in Latin) but this rule doesn't work well on digits since they are
> considered common with other scripts. For instance the input "44사이즈" is kept
> as is even though "사이즈" is part of the dictionary. We should restore the
> original behavior and splits any unknown words if a digit is followed by
> another type.
> This issue was first discovered in
> [https://github.com/elastic/elasticsearch/issues/46365]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[
https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923394#comment-16923394
]
Jim Ferenczi commented on LUCENE-8966:
--
{quote}
Would you consider grouping numbers and (at least some) punctuation together so
that we can preserve decimals and fractions?
{quote}
For complex number grouping and normalization, [~danmuzi] added a
KoreanNumberFilter in https://issues.apache.org/jira/browse/LUCENE-8812
It is identical to the JapaneseNumberFilter excepts that it only detects Korean
hangul numbers. I don't think it handles fractions though but this could be
added if needed.
> KoreanTokenizer should split unknown words on digits
>
>
> Key: LUCENE-8966
> URL: https://issues.apache.org/jira/browse/LUCENE-8966
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8966.patch, LUCENE-8966.patch
>
>
> Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer
> groups characters of unknown words if they belong to the same script or an
> inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the
> rest in Latin) but this rule doesn't work well on digits since they are
> considered common with other scripts. For instance the input "44사이즈" is kept
> as is even though "사이즈" is part of the dictionary. We should restore the
> original behavior and splits any unknown words if a digit is followed by
> another type.
> This issue was first discovered in
> [https://github.com/elastic/elasticsearch/issues/46365]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923378#comment-16923378 ] Mike Sokolov commented on LUCENE-8966: -- Would you consider grouping numbers and (at least some) punctuation together so that we can preserve decimals and fractions? > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch, LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923357#comment-16923357 ] Jim Ferenczi commented on LUCENE-8966: -- Thanks for looking [~thetaphi]. These two private static functions are dead codes that I forgot to remove. The other places use Character.isDigit() consistently. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923335#comment-16923335 ] Uwe Schindler commented on LUCENE-8966: --- Also you are using a private isDigit() at one place, the other place uses Character.isDigit(). This should be consistent. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923332#comment-16923332 ] Uwe Schindler commented on LUCENE-8966: --- isNumber() is dead code? > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[
https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923279#comment-16923279
]
Lucene/Solr QA commented on LUCENE-8966:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} nori in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-8966 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979538/LUCENE-8966.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 78b6530fb26 |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/205/testReport/ |
| modules | C: lucene/analysis/nori U: lucene/analysis/nori |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/205/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> KoreanTokenizer should split unknown words on digits
>
>
> Key: LUCENE-8966
> URL: https://issues.apache.org/jira/browse/LUCENE-8966
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8966.patch
>
>
> Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer
> groups characters of unknown words if they belong to the same script or an
> inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the
> rest in Latin) but this rule doesn't work well on digits since they are
> considered common with other scripts. For instance the input "44사이즈" is kept
> as is even though "사이즈" is part of the dictionary. We should restore the
> original behavior and splits any unknown words if a digit is followed by
> another type.
> This issue was first discovered in
> [https://github.com/elastic/elasticsearch/issues/46365]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8966) KoreanTokenizer should split unknown words on digits
[ https://issues.apache.org/jira/browse/LUCENE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923222#comment-16923222 ] Jim Ferenczi commented on LUCENE-8966: -- Here is a patch that breaks unknown words on digits instead of grouping them with other types. > KoreanTokenizer should split unknown words on digits > > > Key: LUCENE-8966 > URL: https://issues.apache.org/jira/browse/LUCENE-8966 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8966.patch > > > Since https://issues.apache.org/jira/browse/LUCENE-8548 the Korean tokenizer > groups characters of unknown words if they belong to the same script or an > inherited one. This is ok for inputs like Мoscow (with a Cyrillic М and the > rest in Latin) but this rule doesn't work well on digits since they are > considered common with other scripts. For instance the input "44사이즈" is kept > as is even though "사이즈" is part of the dictionary. We should restore the > original behavior and splits any unknown words if a digit is followed by > another type. > This issue was first discovered in > [https://github.com/elastic/elasticsearch/issues/46365] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
