This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 402bfc22a OPENNLP-1885: Address review: checked InvalidFormatException
for malformed models, tag helpers, javadoc throws
add 39ba8ef8f OPENNLP-1885: Address review: validation message style,
shared test fixtures, thread safety wording
No new revisions were added by this update.
Summary of changes:
.../java/opennlp/tools/tokenize/SubwordPiece.java | 2 +-
.../opennlp/tools/tokenize/SubwordTokenizer.java | 3 +-
.../opennlp/tools/tokenize/WordpieceEncoder.java | 12 ++--
.../opennlp/tools/tokenize/WordpieceTokenizer.java | 3 +-
.../tools/tokenize/ReferenceBertPipeline.java | 6 +-
.../tools/tokenize/WordpieceEncoderTest.java | 3 +
opennlp-docs/src/docbkx/tokenizer.xml | 5 +-
.../opennlp/subword/sentencepiece/BpeEncoder.java | 2 +-
.../subword/sentencepiece/ModelProtoReader.java | 2 +-
.../sentencepiece/SentencePieceTokenizer.java | 28 ++++-----
.../subword/sentencepiece/UnigramEncoder.java | 2 +-
.../sentencepiece/SentencePieceAlignmentTest.java | 2 +-
.../sentencepiece/SentencePieceFixtures.java | 67 ++++++++++++++++++++--
.../SentencePieceModelValidationTest.java | 37 +++++++++++-
.../sentencepiece/SentencePieceParityTest.java | 47 +++------------
.../SentencePieceRealModelEvalTest.java | 11 ++--
.../SentencePieceTokenizerSerializationTest.java | 19 +++---
17 files changed, 154 insertions(+), 97 deletions(-)
