This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch feat/GH-1479-tokenizer-character-policy
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 1cb19b326 GH-1479: Configure the tokenizer alphanumeric optimization
with TokenizerCharacterPolicy instead of a regex
add 150196dce Address review nits on the tokenizer character policy
No new revisions were added by this update.
Summary of changes:
README.md | 2 +-
dev/forbidden-regex.txt | 2 +-
.../opennlp/tools/tokenize/TokenizerFactory.java | 72 ++++++++--------------
.../java/opennlp/tools/tokenize/lang/Factory.java | 33 ++++++++--
.../tools/tokenize/TokenizerFactoryTest.java | 35 +++++++++++
opennlp-docs/src/docbkx/tokenizer.xml | 7 ++-
6 files changed, 94 insertions(+), 57 deletions(-)
