This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard bcbb0cab5 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
discard 60a716b69 OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
discard f6464e6df OPENNLP-1885: Document subword tokenization in the manual
discard 95ecac8e6 OPENNLP-1885: Trim residual commentary per review conventions
discard 4c93b267c OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
discard cdef6f20d OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
discard 9b6ecd722 OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
discard 3c0a83902 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
discard 3835e90b0 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
discard 2ccdaf527 OPENNLP-1885: Move the subword contract into opennlp-api
discard cb707afa1 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
discard e797262e8 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 7dae2ca00 OPENNLP-1856: Update JUnit to 6.1.2 (#1172)
add 47367ddf8 OPENNLP-1233: Link the POS and chunk tag references (#1170)
add 64a8439a5 OPENNLP-587: Document the Entity Linker framework (#1169)
add 7cfd7f2dc Change the notification pattern so dev list is not flooded
with emails. (#1176)
add b4f52fa02 OPENNLP-244: Document the BioNLP converter (#1175)
add cf83a1375 OPENNLP-1742: Use the MaxEnt training default for
NameFinderME (#1171)
add 725b1f307 OPENNLP-1868: Add Unicode full case folding to text
normalization (#1138)
add dfd0c94db OPENNLP-1869: Support offset-aware bidirectional emoji and
emoticon expansion and contraction (#1164)
add 9796b1a09 Allowlist based class Instantiation (#1178)
add 4a435d3c3 OPENNLP-1890: Using allowlist based class loading in
StreamFactoryRegistry
add fa79a5243 OPENNLP-1890: Using allowlist based class loading in
StringInterners
add db7adf0a6 OPENNLP-1876: Replace regex with cursor scans in the legacy
CharSequenceNormalizers (#1151)
add 16b82540d Bump actions/setup-java from 5.5.0 to 5.6.0 (#1184)
add 2ae4146b9 OPENNLP-1875: Align whitespace handling with the Unicode
White_Space property (#1150)
add 96ccb0153 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece
inference with exact original-text spans
add 3bfc5846a OPENNLP-1885: Speed up the encode path 2.3x, parity-checked
at every step
add 9af29f1db OPENNLP-1885: Move the subword contract into opennlp-api
add 8052164d1 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased
BertTokenizer into it
add c5df64ac5 OPENNLP-1885: Document the hand-rolled protobuf reader
rationale and refactor trigger
add d9308bbce OPENNLP-1885: Trim commentary and tighten javadoc per review
conventions
add a099f715e OPENNLP-1885: Tighten javadoc to contracts and document
helpers and overrides
add d4f1554e9 OPENNLP-1885: Declare serialVersionUID on
SentencePieceTokenizer
add 14a679dca OPENNLP-1885: Trim residual commentary per review conventions
add d8d5fc89a OPENNLP-1885: Document subword tokenization in the manual
add a8c43c46c OPENNLP-1885: Make the tokenizer graph serializable with
computed UIDs, name the format constants, document every helper
add cc74b8280 OPENNLP-1885: Guard tokenizer deserialization with an
allow-listing ObjectInputFilter
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (bcbb0cab5)
\
N -- N -- N refs/heads/sentencepiece (cc74b8280)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 9 +-
.github/workflows/license.yml | 2 +-
.github/workflows/maven.yml | 8 +-
.github/workflows/publish-snapshots.yml | 2 +-
LICENSE | 2 +-
NOTICE | 3 +
.../main/java/opennlp/tools/util/StringUtil.java | 173 +-
.../java/opennlp/tools/util/WhitespaceMode.java | 134 ++
.../util/normalizer/CharSequenceNormalizer.java | 3 +-
.../util/normalizer/OffsetAwareNormalizer.java | 1 +
.../EntityLinkerManualExamplesTest.java | 177 ++
.../opennlp/tools/util/WhitespaceModeTest.java | 130 ++
.../tools/cmdline/stopword/StopwordFilterTool.java | 5 +-
.../tools/cmdline/TokenNameFinderToolTest.java | 3 +
.../tools/cmdline/StreamFactoryRegistry.java | 27 +-
.../tools/cmdline/StreamFactoryRegistryTest.java | 82 +
.../BioNLP2004NameSampleStreamFactoryTest.java | 24 +
.../tools/langdetect/LanguageDetectorFactory.java | 19 +-
.../java/opennlp/tools/namefind/NameFinderME.java | 4 +-
.../sentdetect/DefaultSDContextGenerator.java | 6 +-
.../tools/stopword/DictionaryStopwordFilter.java | 9 +-
.../tokenize/DefaultTokenContextGenerator.java | 6 +
.../tools/util/featuregen/GeneratorFactory.java | 45 +-
.../tools/util/jvm/CHMStringDeduplicator.java | 2 +-
.../opennlp/tools/util/jvm/CHMStringInterner.java | 2 +-
.../opennlp/tools/util/jvm/HMStringInterner.java | 2 +-
.../opennlp/tools/util/jvm/JvmStringInterner.java | 2 +-
.../opennlp/tools/util/jvm/NoOpStringInterner.java | 2 +-
.../opennlp/tools/util/jvm/StringInterners.java | 12 +-
.../AccentFoldCharSequenceNormalizer.java | 6 +-
.../AggregateCharSequenceNormalizer.java | 4 +
.../opennlp/tools/util/normalizer/AsciiChars.java | 77 +
.../normalizer/CaseFoldCharSequenceNormalizer.java | 6 +-
.../ConfusableSkeletonCharSequenceNormalizer.java | 6 +-
.../opennlp/tools/util/normalizer/Dimension.java | 20 +-
.../normalizer/EmojiCharSequenceNormalizer.java | 8 +
.../tools/util/normalizer/EmojiEmoticons.java | 442 +++++
.../EmojiToEmoticonCharSequenceNormalizer.java | 63 +
.../EmoticonToEmojiCharSequenceNormalizer.java | 64 +
.../FullCaseFoldCharSequenceNormalizer.java | 193 +++
.../util/normalizer/NfcCharSequenceNormalizer.java | 6 +-
.../normalizer/NfkcCharSequenceNormalizer.java | 6 +-
.../normalizer/NumberCharSequenceNormalizer.java | 33 +-
.../normalizer/ShrinkCharSequenceNormalizer.java | 128 +-
.../SocialMediaCharSequenceNormalizer.java | 306 ++++
.../tools/util/normalizer/TermAnalyzer.java | 56 +
.../tools/util/normalizer/TextNormalizer.java | 80 +-
.../normalizer/TwitterCharSequenceNormalizer.java | 55 -
.../util/normalizer/UrlCharSequenceNormalizer.java | 192 ++-
.../opennlp/tools/util/normalizer/CaseFolding.txt | 1682 ++++++++++++++++++++
.../tools/util/normalizer/emoji-emoticons.txt | 90 ++
.../opennlp/tools/namefind/NameFinderMETest.java | 3 +
.../tools/namefind/NameFinderMEWithDatesTest.java | 18 +-
.../SentenceDetectorMESpanMappingTest.java | 14 +
.../tokenize/DefaultTokenContextGeneratorTest.java | 80 +
.../tools/tokenize/SimpleTokenizerTest.java | 46 +
.../tools/tokenize/WhitespaceTokenizerTest.java | 46 +
.../util/featuregen/GeneratorFactoryTest.java | 46 +-
.../tools/util/jvm/StringInternersTest.java | 74 +
.../normalizer/AlignedNormalizerPipelineTest.java | 8 +-
.../CharSequenceNormalizerContractTest.java | 88 +
.../util/normalizer/CharacterizationInputs.java | 51 +
.../tools/util/normalizer/EmojiEmoticonsTest.java | 181 +++
.../EmojiToEmoticonCharSequenceNormalizerTest.java | 164 ++
.../EmoticonToEmojiCharSequenceNormalizerTest.java | 135 ++
.../FullCaseFoldCharSequenceNormalizerTest.java | 275 ++++
...CharSequenceNormalizerCharacterizationTest.java | 102 ++
...CharSequenceNormalizerCharacterizationTest.java | 166 ++
...CharSequenceNormalizerCharacterizationTest.java | 198 +++
.../tools/util/normalizer/TermAnalyzerTest.java | 76 +
.../tools/util/normalizer/TextNormalizerTest.java | 29 +-
...CharSequenceNormalizerCharacterizationTest.java | 204 +++
opennlp-distr/src/main/readme/LICENSE | 1 +
opennlp-distr/src/main/readme/NOTICE | 3 +
opennlp-docs/src/docbkx/chunker.xml | 7 +-
opennlp-docs/src/docbkx/cli.xml | 8 +
opennlp-docs/src/docbkx/corpora.xml | 74 +
opennlp-docs/src/docbkx/entitylinker.xml | 213 +++
opennlp-docs/src/docbkx/introduction.xml | 50 +
opennlp-docs/src/docbkx/normalizer.xml | 35 +-
opennlp-docs/src/docbkx/opennlp.xml | 1 +
opennlp-docs/src/docbkx/tokenizer.xml | 5 +
.../opennlp/tools/eval/SourceForgeModelEval.java | 19 +
.../spellcheck/cmdline/CorrectTextTool.java | 6 +-
.../SpellCheckingCharSequenceNormalizer.java | 211 ++-
.../stream/SpellCorrectingObjectStream.java | 31 +-
.../stream/SpellCorrectingTokenStream.java | 67 +-
.../java/opennlp/spellcheck/symspell/SymSpell.java | 9 +-
.../SpellCheckingCharSequenceNormalizerTest.java | 122 +-
.../stream/SpellCorrectingObjectStreamTest.java | 16 +-
.../stream/SpellCorrectingTokenStreamTest.java | 47 +
.../spellcheck/symspell/SymSpellCompoundTest.java | 64 +
.../java/opennlp/uima/normalizer/NumberUtil.java | 26 +-
.../opennlp/uima/normalizer/NumberUtilTest.java | 32 +
.../java/opennlp/tools/util/StringUtilTest.java | 623 +++++++-
.../TwitterCharSequenceNormalizerTest.java | 61 -
pom.xml | 2 +-
src/license/NOTICE.template | 3 +
98 files changed, 7744 insertions(+), 415 deletions(-)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/util/WhitespaceMode.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/entitylinker/EntityLinkerManualExamplesTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/util/WhitespaceModeTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/cmdline/StreamFactoryRegistryTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/AsciiChars.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEmoticons.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiToEmoticonCharSequenceNormalizer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmoticonToEmojiCharSequenceNormalizer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/SocialMediaCharSequenceNormalizer.java
delete mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TwitterCharSequenceNormalizer.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/CaseFolding.txt
create mode 100644
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-emoticons.txt
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DefaultTokenContextGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/jvm/StringInternersTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/CharSequenceNormalizerContractTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/CharacterizationInputs.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiEmoticonsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiToEmoticonCharSequenceNormalizerTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmoticonToEmojiCharSequenceNormalizerTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizerTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NumberCharSequenceNormalizerCharacterizationTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/ShrinkCharSequenceNormalizerCharacterizationTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/SocialMediaCharSequenceNormalizerCharacterizationTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizerCharacterizationTest.java
create mode 100644 opennlp-docs/src/docbkx/entitylinker.xml
delete mode 100644
opennlp-tools/src/test/java/opennlp/tools/util/normalizer/TwitterCharSequenceNormalizerTest.java
