This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 6d40bc071 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
add 3405694f5 OPENNLP-1885: Cite the SentencePiece usage example test in
the manual
add e82aa10fb OPENNLP-1885: Align null contracts and annotations with the
review conventions
No new revisions were added by this update.
Summary of changes:
.../opennlp/tools/tokenize/WordpieceEncoder.java | 21 ++++++-
opennlp-docs/src/docbkx/tokenizer.xml | 9 +--
.../sentencepiece/SentencePieceTokenizer.java | 2 +
.../subword/sentencepiece/PieceTrieTest.java | 3 +-
.../SentencePieceUsageExampleTest.java | 69 ++++++++++++++++++++++
5 files changed, 96 insertions(+), 8 deletions(-)
create mode 100644
opennlp-extensions/opennlp-subword/src/test/java/opennlp/subword/sentencepiece/SentencePieceUsageExampleTest.java
