[
https://issues.apache.org/jira/browse/OPENNLP-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111071#comment-18111071
]
ASF GitHub Bot commented on OPENNLP-1885:
-----------------------------------------
krickert opened a new pull request, #178:
URL: https://github.com/apache/opennlp-addons/pull/178
## Summary
- implements the OpenNLP `SubwordTokenizer` API with pure Java SentencePiece
inference
- loads SentencePiece `.model` files and supports unigram and byte-pair
models
- preserves exact UTF-16 spans into the caller's original text after model
normalization
- exposes vocabulary lookup, piece scores, algorithm type, and aligned
normalization
- adds generated test models, independent parity fixtures, usage
documentation, and 66 tests
No runtime model, native library, or third-party training data is bundled.
This is the example add-on contribution described by
apache/opennlp-addons#177.
## Dependencies
- apache/opennlp-addons#177 prepares the add-ons canary, manual, and reactor
- apache/opennlp#1165 supplies `SubwordTokenizer`, `SubwordPiece`, and the
shared offset contract
The PR targets `OPENNLP-1224-canary-addon` because GitHub requires the base
branch to exist in the Apache repository. Its diff includes #177 until that PR
merges, then narrows to the SentencePiece commit.
## Validation
- `mvn clean verify -Dopennlp.forkCount=1`: all seven reactor projects passed
- SentencePiece: 66 tests passed, with one opt-in real-model evaluation
skipped
- `mvn -N clean site`: 9 HTML pages generated
- local links and fragments passed
- RAT reported zero unapproved files
- all six serializable implementation classes match their computed
`serialver` values
> Pure-Java SentencePiece inference with exact original-text spans
> -----------------------------------------------------------------
>
> Key: OPENNLP-1885
> URL: https://issues.apache.org/jira/browse/OPENNLP-1885
> Project: OpenNLP
> Issue Type: New Feature
> Components: Sentence Detector
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6, 3.0.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Adds a new opennlp-extensions/opennlp-subword module implementing
> SentencePiece model inference purely in Java: the model file reader
> (hand-written protobuf wire parsing, no new dependency), the model-embedded
> normalizer (precompiled character map over the double-array trie format,
> whitespace collapsing and escaping, word-boundary marker), unigram best-path
> segmentation, BPE agenda merging, byte fallback, and user-defined symbol
> handling. The public contract is SubwordTokenizer/SubwordPiece; every piece
> reports the exact UTF-16 span of the caller's original text it came from, and
> the model normalizer is also exposed as an OffsetAwareNormalizer producing
> AlignedText.
> Parity with the reference implementation is asserted, not assumed. Five tiny
> trained models are bundled with fixtures generated by the sentencepiece
> Python package (unigram, unigram with byte fallback, BPE, identity
> normalization, whitespace-as-suffix), checked piece for piece, id for id,
> span for span, plus each model's embedded self-test samples. An opt-in test
> (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded
> models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly,
> including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture
> regeneration scripts live in the test resources.
> Measured single-thread throughput on the t5-small vocabulary is about 2.8M
> pieces/s; the native reference is about 1.6x faster. The value here is zero
> native dependencies, one shareable thread-safe instance, and exact
> original-text offsets. A non-breaking performance follow-up with identified
> wins is planned separately.
> JIRA to follow; title gets the key once filed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
