krickert opened a new pull request, #1165: URL: https://github.com/apache/opennlp/pull/1165
Adds a new opennlp-extensions/opennlp-subword module implementing SentencePiece model inference purely in Java: the model file reader (hand-written protobuf wire parsing, no new dependency), the model-embedded normalizer (precompiled character map over the double-array trie format, whitespace collapsing and escaping, word-boundary marker), unigram best-path segmentation, BPE agenda merging, byte fallback, and user-defined symbol handling. The public contract is SubwordTokenizer/SubwordPiece; every piece reports the exact UTF-16 span of the caller's original text it came from, and the model normalizer is also exposed as an OffsetAwareNormalizer producing AlignedText. Parity with the reference implementation is asserted, not assumed. Five tiny trained models are bundled with fixtures generated by the sentencepiece Python package (unigram, unigram with byte fallback, BPE, identity normalization, whitespace-as-suffix), checked piece for piece, id for id, span for span, plus each model's embedded self-test samples. An opt-in test (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly, including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture regeneration scripts live in the test resources. Measured single-thread throughput on the t5-small vocabulary is about 2.8M pieces/s; the native reference is about 1.6x faster. The value here is zero native dependencies, one shareable thread-safe instance, and exact original-text offsets. A non-breaking performance follow-up with identified wins is planned separately. JIRA to follow; title gets the key once filed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
