[
https://issues.apache.org/jira/browse/OPENNLP-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121754#comment-18121754
]
ASF GitHub Bot commented on OPENNLP-1934:
-----------------------------------------
krickert commented on PR #1349:
URL: https://github.com/apache/opennlp/pull/1349#issuecomment-5942159649
This can wait for 3.x, but leaving open for now since it is ready.
The there's follow-up code I'll see about in another time, it deprecates the
tokenizer's Pattern constructors instead of replacing them. There's no rush in
this, but it'll help performance and move us toward being 100% regex free.
> RegEx removal part 6: evaluate the tokenizer alphanumeric pattern as a
> character set
> ------------------------------------------------------------------------------------
>
> Key: OPENNLP-1934
> URL: https://issues.apache.org/jira/browse/OPENNLP-1934
> Project: OpenNLP
> Issue Type: Improvement
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Decision: the alphanumeric pattern stays a {{Pattern}} in
> {{TokenizerFactory}} and {{lang.Factory}}. It is written to the model
> manifest as a regex string and read back with {{Pattern.compile}}, so the
> type is part of the model contract and every existing model carries one.
> What changes is the hot path. A package-private {{AlphaNumericCheck}} reads a
> pattern of the shape {{^[...]+$}} whose class holds only literal characters
> and simple ranges, which covers every built-in language default, into a
> character set and tests tokens by lookup. Any other pattern, and any pattern
> compiled with flags, is still matched by the engine, so the result equals
> {{pattern.matcher(token).matches()}} in all cases. {{TokenizerME}} and
> {{TokSpanEventStream}} hold the check instead of the {{Pattern}}.
> Verification: the test compares the check with the regex for all built-in
> defaults and a set of custom patterns over the same token table, and pins
> which shapes run as a set lookup. Full runtime suite green (3,624 tests).
> The five tokenizer classes move from "still determining" to the exempt list
> in the epic, with this ticket as the reason.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
