[
https://issues.apache.org/jira/browse/OPENNLP-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121388#comment-18121388
]
ASF GitHub Bot commented on OPENNLP-1934:
-----------------------------------------
krickert opened a new pull request, #1349:
URL: https://github.com/apache/opennlp/pull/1349
Adds `TokenizerCharacterPolicy` as a first step toward replacing the
tokenizer’s configurable alphanumeric regex.
The policy defines which code points count as letters, digits and combining
marks. `test(CharSequence)` checks candidates by code point, supports
supplementary characters, and rejects malformed UTF-16. The `ascii()` preset
matches the current default pattern for ASCII input. Policies with empty or
overlapping sets are rejected.
This PR adds the policy class without changing tokenizer behavior. Training,
inference, model persistence and Unicode presets are follow-up work in a
stacked draft. The existing regex configuration continues to work.
Tests cover character and combining-mark rules, presets, malformed input,
null handling and the manual example. The tokenizer chapter documents the new
class.
[OPENNLP-1934](https://issues.apache.org/jira/browse/OPENNLP-1934)
Continues #1325 on an apache branch, rebased onto main.
> RegEx removal part 6: evaluate the tokenizer alphanumeric pattern as a
> character set
> ------------------------------------------------------------------------------------
>
> Key: OPENNLP-1934
> URL: https://issues.apache.org/jira/browse/OPENNLP-1934
> Project: OpenNLP
> Issue Type: Improvement
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Decision: the alphanumeric pattern stays a {{Pattern}} in
> {{TokenizerFactory}} and {{lang.Factory}}. It is written to the model
> manifest as a regex string and read back with {{Pattern.compile}}, so the
> type is part of the model contract and every existing model carries one.
> What changes is the hot path. A package-private {{AlphaNumericCheck}} reads a
> pattern of the shape {{^[...]+$}} whose class holds only literal characters
> and simple ranges, which covers every built-in language default, into a
> character set and tests tokens by lookup. Any other pattern, and any pattern
> compiled with flags, is still matched by the engine, so the result equals
> {{pattern.matcher(token).matches()}} in all cases. {{TokenizerME}} and
> {{TokSpanEventStream}} hold the check instead of the {{Pattern}}.
> Verification: the test compares the check with the regex for all built-in
> defaults and a set of custom patterns over the same token table, and pins
> which shapes run as a set lookup. Full runtime suite green (3,624 tests).
> The five tokenizer classes move from "still determining" to the exempt list
> in the epic, with this ticket as the reason.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
