This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard a42d40f2a OPENNLP-1885: Address review: checked InvalidFormatException
for malformed models, tag helpers, javadoc throws
discard e82aa10fb OPENNLP-1885: Align null contracts and annotations with the
review conventions
discard 3405694f5 OPENNLP-1885: Cite the SentencePiece usage example test in
the manual
omit 6d40bc071 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
omit 0f1e2c832 OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
omit 7e4570cbc OPENNLP-1885: Document subword tokenization in the manual
omit d4eb7dd2d OPENNLP-1885: Trim residual commentary per review conventions
omit 141df4789 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
omit 5c1e8c101 OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
omit 37108ceda OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
omit e7adb8e31 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
omit 09d77f123 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
omit 935fcc28d OPENNLP-1885: Move the subword contract into opennlp-api
omit 3e3b6e67b OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
omit 6947fb4f3 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add ef0a354cf OPENNLP-1883: Make stemmers thread-safe, add StemmerFactory
and per-thread stem caching (#1163)
add 55eaaaaf6 Bump actions/checkout from 7.0.0 to 7.0.1 (#1189)
add 9951bfd0a OPENNLP-1870: Add opt-in emoji annotations for NameFinder,
Doccat, and Sentiment (#1177)
add 8eef045b6 Minor: Regenerated NOTICE File for
9951bfd0ae0aee8ca77eb436702e5544fbea2afc
add 66526b528 [maven-release-plugin] prepare release opennlp-3.0.0-M5
add a8642301f [maven-release-plugin] prepare for next development iteration
add e746074b9 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 534293c76 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
add 6948237aa OPENNLP-1885: Move the subword contract into opennlp-api
add c0adc1e56 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
add 141297de9 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
add 4a5c52bf2 OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
add 3f0811da2 OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
add a77702bd3 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
add 93ce37741 OPENNLP-1885: Trim residual commentary per review conventions
add 28340a09c OPENNLP-1885: Document subword tokenization in the manual
add 9c84dfadb OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
add bc31242c8 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
add f5ae27ff1 OPENNLP-1885: Cite the SentencePiece usage example test in
the manual
add 5e3ae5add OPENNLP-1885: Align null contracts and annotations with the
review conventions
add 402bfc22a OPENNLP-1885: Address review: checked InvalidFormatException
for malformed models, tag helpers, javadoc throws
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a42d40f2a)
\
N -- N -- N refs/heads/sentencepiece (402bfc22a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/allowlist-check.yml | 2 +-
.github/workflows/license.yml | 2 +-
.github/workflows/maven.yml | 8 +-
.github/workflows/publish-snapshots.yml | 2 +-
LICENSE | 8 +-
NOTICE | 9 +
.../main/java/opennlp/tools/stemmer/Stemmer.java | 25 +-
.../stemmer/{Stemmer.java => StemmerFactory.java} | 13 +-
opennlp-core/opennlp-runtime/BENCHMARKS.md | 84 ++++-
.../tools/stemmer/CachingStemmerBenchmark.java | 199 +++++++++++
.../snowball/SnowballStemmerBenchmark.java} | 106 +++---
.../tools/doccat/EmojiFeatureGenerator.java | 82 +++++
.../sentiment/EmojiSentimentContextGenerator.java | 87 +++++
.../tools/sentiment/EmojiSentimentFactory.java} | 46 +--
.../java/opennlp/tools/stemmer/CachingStemmer.java | 152 +++++++++
.../opennlp/tools/stemmer/DelegatingStemmer.java | 70 ++++
.../java/opennlp/tools/stemmer/PorterStemmer.java | 7 +
.../PorterStemmerFactory.java} | 16 +-
.../java/opennlp/tools/stemmer/SharingStemmer.java | 67 ++++
.../tools/stemmer/snowball/SnowballStemmer.java | 135 +++++---
.../stemmer/snowball/SnowballStemmerFactory.java | 109 ++++++
.../EmojiAnnotationFeatureGenerator.java | 83 +++++
...=> EmojiAnnotationFeatureGeneratorFactory.java} | 24 +-
.../tools/util/normalizer/EmojiAnnotation.java | 197 +++++++++++
.../tools/util/normalizer/EmojiAnnotationJoin.java | 51 +++
.../tools/util/normalizer/EmojiAnnotations.java | 256 ++++++++++++++
.../tools/util/normalizer/EmojiAnnotator.java | 197 +++++++++++
.../tools/util/normalizer/EmojiCategory.java | 62 ++++
.../tools/util/normalizer/EmojiEntityType.java | 57 ++++
.../opennlp/tools/util/normalizer/EmojiFlags.java | 227 +++++++++++++
.../util/normalizer/NormalizationProfile.java | 23 +-
.../tools/util/normalizer/TermAnalyzer.java | 43 ++-
.../tools/util/normalizer/emoji-annotations.txt | 212 ++++++++++++
.../tools/doccat/EmojiFeatureGeneratorTest.java | 80 +++++
.../EmojiSentimentContextGeneratorTest.java | 72 ++++
.../opennlp/tools/stemmer/CachingStemmerTest.java | 139 ++++++++
.../opennlp/tools/stemmer/StemmerFactoryTest.java | 376 +++++++++++++++++++++
.../stemmer/StemmerFactoryUsageExampleTest.java | 62 ++++
.../EmojiAnnotationFeatureGeneratorTest.java | 127 +++++++
.../util/normalizer/EmojiAnnotationsTest.java | 256 ++++++++++++++
.../tools/util/normalizer/EmojiAnnotatorTest.java | 176 ++++++++++
.../normalizer/EmojiAnnotatorUsageExampleTest.java | 65 ++++
.../tools/util/normalizer/EmojiFlagsTest.java | 147 ++++++++
.../util/normalizer/NormalizationProfilesTest.java | 23 ++
.../tools/util/normalizer/NormalizerTestUtil.java | 29 +-
.../tools/util/normalizer/TermAnalyzerTest.java | 28 +-
opennlp-distr/src/main/readme/LICENSE | 5 +-
opennlp-distr/src/main/readme/NOTICE | 9 +
opennlp-docs/src/docbkx/normalizer.xml | 67 +++-
opennlp-docs/src/docbkx/opennlp.xml | 1 +
opennlp-docs/src/docbkx/stemmer.xml | 72 ++++
.../tools/eval/MultiThreadedStemmerEval.java | 175 ++++++++++
src/license/NOTICE.template | 9 +
53 files changed, 4375 insertions(+), 204 deletions(-)
copy opennlp-api/src/main/java/opennlp/tools/stemmer/{Stemmer.java =>
StemmerFactory.java} (68%)
create mode 100644
opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/stemmer/CachingStemmerBenchmark.java
copy
opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/{tokenize/TokenizerMEBenchmark.java
=> stemmer/snowball/SnowballStemmerBenchmark.java} (54%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
copy
opennlp-core/{opennlp-ml/opennlp-ml-commons/src/main/java/opennlp/tools/ml/model/ObjectDataReader.java
=>
opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java}
(50%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/CachingStemmer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/DelegatingStemmer.java
copy
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/{util/jvm/NoOpStringInterner.java
=> stemmer/PorterStemmerFactory.java} (70%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmerFactory.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
copy
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/{CharacterNgramFeatureGeneratorFactory.java
=> EmojiAnnotationFeatureGeneratorFactory.java} (60%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/doccat/EmojiFeatureGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentiment/EmojiSentimentContextGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/CachingStemmerTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/StemmerFactoryTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/StemmerFactoryUsageExampleTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorUsageExampleTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiFlagsTest.java
copy
opennlp-api/src/main/java/opennlp/tools/util/normalizer/CharSequenceNormalizer.java
=>
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NormalizerTestUtil.java
(57%)
create mode 100644 opennlp-docs/src/docbkx/stemmer.xml
create mode 100644
opennlp-eval-tests/src/test/java/opennlp/tools/eval/MultiThreadedStemmerEval.java
