This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit cc74b8280 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
omit a8c43c46c OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
omit d8d5fc89a OPENNLP-1885: Document subword tokenization in the manual
omit 14a679dca OPENNLP-1885: Trim residual commentary per review conventions
omit d4f1554e9 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
omit a099f715e OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
omit d9308bbce OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
omit c5df64ac5 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
omit 8052164d1 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
omit 9af29f1db OPENNLP-1885: Move the subword contract into opennlp-api
omit 3bfc5846a OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
omit 96ccb0153 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 728186acc OPENNLP-1891: Harden Deserialization in BaseModel (#1185)
add 011195f34 OPENNLP-1878: Non-breaking performance follow-up for
normalization and tokenization hot paths (#1161)
add 6038bf503 OPENNLP-1892: Precompile Tag patterns to avoid repeated
pattern compilation (#1187)
add 6947fb4f3 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 3e3b6e67b OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
add 935fcc28d OPENNLP-1885: Move the subword contract into opennlp-api
add 09d77f123 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
add e7adb8e31 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
add 37108ceda OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
add 5c1e8c101 OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
add 141df4789 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
add d4eb7dd2d OPENNLP-1885: Trim residual commentary per review conventions
add 7e4570cbc OPENNLP-1885: Document subword tokenization in the manual
add 0f1e2c832 OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
add 6d40bc071 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cc74b8280)
\
N -- N -- N refs/heads/sentencepiece (6d40bc071)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../main/java/opennlp/tools/util/StringUtil.java | 8 +
.../opennlp/tools/util/normalizer/Alignment.java | 68 +++++-
.../opennlp/tools/util/normalizer/CharClass.java | 268 +++++++++++++--------
.../opennlp/tools/util/normalizer/CodePoints.java | 92 +++++++
.../tools/util/normalizer/AlignmentTest.java | 32 +++
.../tools/util/normalizer/CharClassTest.java | 91 ++++++-
.../tools/util/normalizer/CodePointsTest.java | 138 +++++++++++
.../src/main/java/opennlp/dl/AbstractDL.java | 4 +-
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 12 +-
.../java/opennlp/dl/namefinder/NameFinderDL.java | 10 +-
.../parser/lang/es/AncoraSpanishHeadRules.java | 53 ++--
.../java/opennlp/tools/util/model/BaseModel.java | 42 +++-
.../AlignedAggregateCharSequenceNormalizer.java | 5 +-
.../opennlp/tools/util/normalizer/Confusables.java | 109 ++++++---
.../normalizer/DigitCharSequenceNormalizer.java | 15 +-
.../parser/lang/es/AncoraSpanishHeadRulesTest.java | 72 ++++++
.../tools/tokenize/uax29/WordSegmenterTest.java | 7 +
.../opennlp/tools/util/model/BaseModelTest.java | 91 +++++++
.../normalizer/AlignedNormalizerPipelineTest.java | 17 ++
.../tools/util/normalizer/ConfusablesTest.java | 49 ++++
.../util/normalizer/SetBasedNormalizerTest.java | 7 +
21 files changed, 1017 insertions(+), 173 deletions(-)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/util/normalizer/CodePoints.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/util/normalizer/CodePointsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRulesTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/model/BaseModelTest.java
