[
https://issues.apache.org/jira/browse/OPENNLP-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121390#comment-18121390
]
ASF GitHub Bot commented on OPENNLP-1934:
-----------------------------------------
krickert closed pull request #1325: OPENNLP-1934: Add explicit tokenizer
character policies
URL: https://github.com/apache/opennlp/pull/1325
> RegEx removal part 6: evaluate the tokenizer alphanumeric pattern as a
> character set
> ------------------------------------------------------------------------------------
>
> Key: OPENNLP-1934
> URL: https://issues.apache.org/jira/browse/OPENNLP-1934
> Project: OpenNLP
> Issue Type: Improvement
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Decision: the alphanumeric pattern stays a {{Pattern}} in
> {{TokenizerFactory}} and {{lang.Factory}}. It is written to the model
> manifest as a regex string and read back with {{Pattern.compile}}, so the
> type is part of the model contract and every existing model carries one.
> What changes is the hot path. A package-private {{AlphaNumericCheck}} reads a
> pattern of the shape {{^[...]+$}} whose class holds only literal characters
> and simple ranges, which covers every built-in language default, into a
> character set and tests tokens by lookup. Any other pattern, and any pattern
> compiled with flags, is still matched by the engine, so the result equals
> {{pattern.matcher(token).matches()}} in all cases. {{TokenizerME}} and
> {{TokSpanEventStream}} hold the check instead of the {{Pattern}}.
> Verification: the test compares the check with the regex for all built-in
> defaults and a set of custom patterns over the same token table, and pins
> which shapes run as a set lookup. Full runtime suite green (3,624 tests).
> The five tokenizer classes move from "still determining" to the exempt list
> in the epic, with this ticket as the reason.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
