Kristian Rickert created OPENNLP-1934:
-----------------------------------------
Summary: RegEx removal part 6: evaluate the tokenizer alphanumeric
pattern as a character set
Key: OPENNLP-1934
URL: https://issues.apache.org/jira/browse/OPENNLP-1934
Project: OpenNLP
Issue Type: Improvement
Components: Tokenizer
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Decision: the alphanumeric pattern stays a {{Pattern}} in {{TokenizerFactory}}
and {{lang.Factory}}. It is written to the model manifest as a regex string and
read back with {{Pattern.compile}}, so the type is part of the model contract
and every existing model carries one.
What changes is the hot path. A package-private {{AlphaNumericCheck}} reads a
pattern of the shape {{^[...]+$}} whose class holds only literal characters and
simple ranges, which covers every built-in language default, into a character
set and tests tokens by lookup. Any other pattern, and any pattern compiled
with flags, is still matched by the engine, so the result equals
{{pattern.matcher(token).matches()}} in all cases. {{TokenizerME}} and
{{TokSpanEventStream}} hold the check instead of the {{Pattern}}.
Verification: the test compares the check with the regex for all built-in
defaults and a set of custom patterns over the same token table, and pins which
shapes run as a set lookup. Full runtime suite green (3,624 tests).
The five tokenizer classes move from "still determining" to the exempt list in
the epic, with this ticket as the reason.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
