krickert opened a new issue, #1479: URL: https://github.com/apache/opennlp/issues/1479
#1349 added TokenizerCharacterPolicy, but TokenizerME, TokSpanEventStream, TokenizerFactory and the language Factory still evaluate the alphanumeric pattern with java.util.regex. They are the largest group of members exempt from the regex check added in #1348. Solution design: The tokenizer and its event stream would use TokenizerCharacterPolicy. A pattern stored in an existing model manifest is read once at load and mapped to the matching policy; a pattern that has no policy is rejected when the model loads. Should the Pattern-based constructors and getters would go away in 3.0, the tokenizer package no longer needs @SuppressForbidden. AI suggestion on testing: > The bundled language policies should be tested with their old regular expressions as the oracle, and TokenizerME output pinned for each, so tokenization does not change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
