This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 39ba8ef8f OPENNLP-1885: Address review: validation message style,
shared test fixtures, thread safety wording
omit 402bfc22a OPENNLP-1885: Address review: checked InvalidFormatException
for malformed models, tag helpers, javadoc throws
omit 5e3ae5add OPENNLP-1885: Align null contracts and annotations with the
review conventions
omit f5ae27ff1 OPENNLP-1885: Cite the SentencePiece usage example test in
the manual
omit bc31242c8 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
omit 9c84dfadb OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
omit 28340a09c OPENNLP-1885: Document subword tokenization in the manual
omit 93ce37741 OPENNLP-1885: Trim residual commentary per review conventions
omit a77702bd3 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
omit 3f0811da2 OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
omit 4a5c52bf2 OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
omit 141297de9 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
omit c0adc1e56 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
omit 6948237aa OPENNLP-1885: Move the subword contract into opennlp-api
omit 534293c76 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
omit e746074b9 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 71e1fd94f OPENNLP-1898: Address the OPENNLP-1876 review follow-ups
(#1196)
add 93fe6798a OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add d97054720 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
add 6c8195b96 OPENNLP-1885: Move the subword contract into opennlp-api
add b6488a814 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
add 0a70edc48 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
add 8d7496b0a OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
add d86892712 OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
add 92a3e992d OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
add 58bfc53b6 OPENNLP-1885: Trim residual commentary per review conventions
add 5dbb92a95 OPENNLP-1885: Document subword tokenization in the manual
add 865f48550 OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
add a8597ba91 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
add 6ad917e9f OPENNLP-1885: Cite the SentencePiece usage example test in
the manual
add 0aad3dd6a OPENNLP-1885: Align null contracts and annotations with the
review conventions
add 2d14e49e8 OPENNLP-1885: Address review: checked InvalidFormatException
for malformed models, tag helpers, javadoc throws
add 1c0af89a0 OPENNLP-1885: Address review: validation message style,
shared test fixtures, thread safety wording
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (39ba8ef8f)
\
N -- N -- N refs/heads/sentencepiece (1c0af89a0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
...CharSequenceNormalizerCharacterizationTest.java | 6 ++-
...CharSequenceNormalizerCharacterizationTest.java | 11 +++--
...CharSequenceNormalizerCharacterizationTest.java | 21 +++++----
...CharSequenceNormalizerCharacterizationTest.java | 54 +++++++++++++---------
.../SpellCheckingCharSequenceNormalizer.java | 9 +++-
.../stream/SpellCorrectingTokenStream.java | 8 ++--
6 files changed, 68 insertions(+), 41 deletions(-)
