[
https://issues.apache.org/jira/browse/OPENNLP-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693714#comment-17693714
]
ASF GitHub Bot commented on OPENNLP-141:
----------------------------------------
kinow commented on PR #506:
URL: https://github.com/apache/opennlp/pull/506#issuecomment-1445471012
>For the records: I checked Italian, yet it seems they have no special
characters in their alphabet.
They should have some accents. I don't speak Italian but we had **a lot** of
soap operas about Italian immigrants, and "più" always appeared in
writing/speaking. Some Google Translate gives: “nor the most beautiful nor the
most ugly” → “né il più bello né il più brutto” —
https://it.wikipedia.org/wiki/Alfabeto_italiano
> I thought of Spanish, yet could not find a valid(ated) pattern. Do you
know of any proven es regex?
Good question. I'm still studying Spanish, so good opportunity for me to
learn more. Give me some time and I will find one (will spend some time
searching & reading about alphabet/letters/etc.).
> Tokenizers alpha numeric optimization only recognizes a-z as alpha chars
> ------------------------------------------------------------------------
>
> Key: OPENNLP-141
> URL: https://issues.apache.org/jira/browse/OPENNLP-141
> Project: OpenNLP
> Issue Type: Bug
> Components: Tokenizer
> Affects Versions: tools-1.5.0-sourceforge
> Reporter: Jörn Kottmann
> Assignee: Martin Wiesner
> Priority: Minor
>
> The Tokenizer has an optimization which skips tokens which are only made of
> numerics or alpha chars. In foreign languages the alpha chars contain umlauts
> and other letters which are not included in the a-z range.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
