krickert opened a new pull request, #178: URL: https://github.com/apache/opennlp-addons/pull/178
## Summary - implements the OpenNLP `SubwordTokenizer` API with pure Java SentencePiece inference - loads SentencePiece `.model` files and supports unigram and byte-pair models - preserves exact UTF-16 spans into the caller's original text after model normalization - exposes vocabulary lookup, piece scores, algorithm type, and aligned normalization - adds generated test models, independent parity fixtures, usage documentation, and 66 tests No runtime model, native library, or third-party training data is bundled. This is the example add-on contribution described by apache/opennlp-addons#177. ## Dependencies - apache/opennlp-addons#177 prepares the add-ons canary, manual, and reactor - apache/opennlp#1165 supplies `SubwordTokenizer`, `SubwordPiece`, and the shared offset contract The PR targets `OPENNLP-1224-canary-addon` because GitHub requires the base branch to exist in the Apache repository. Its diff includes #177 until that PR merges, then narrows to the SentencePiece commit. ## Validation - `mvn clean verify -Dopennlp.forkCount=1`: all seven reactor projects passed - SentencePiece: 66 tests passed, with one opt-in real-model evaluation skipped - `mvn -N clean site`: 9 HTML pages generated - local links and fragments passed - RAT reported zero unapproved files - all six serializable implementation classes match their computed `serialver` values -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
