[
https://issues.apache.org/jira/browse/OPENNLP-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693670#comment-17693670
]
ASF GitHub Bot commented on OPENNLP-141:
----------------------------------------
rzo1 commented on code in PR #506:
URL: https://github.com/apache/opennlp/pull/506#discussion_r1118111204
##########
opennlp-tools/src/main/java/opennlp/tools/tokenize/lang/Factory.java:
##########
@@ -25,24 +25,45 @@
public class Factory {
- public static final String DEFAULT_ALPHANUMERIC = "^[A-Za-z0-9]+$";
+ public static final Pattern DEFAULT_ALPHANUMERIC =
Pattern.compile("^[A-Za-z0-9]+$");
Review Comment:
I am fine with the reasoning for keeping it public atm, but would prefer to
make it private in the long run (maybe via the @Deprecated approach as with
`alphaNumeric` before). 🙃
> Tokenizers alpha numeric optimization only recognizes a-z as alpha chars
> ------------------------------------------------------------------------
>
> Key: OPENNLP-141
> URL: https://issues.apache.org/jira/browse/OPENNLP-141
> Project: OpenNLP
> Issue Type: Bug
> Components: Tokenizer
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Martin Wiesner
> Priority: Minor
>
> The Tokenizer has an optimization which skips tokens which are only made of
> numerics or alpha chars. In foreign languages the alpha chars contain umlauts
> and other letters which are not included in the a-z range.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
