This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1894-lattice-cjk
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard 65579e9cd OPENNLP-1894: Trim parsed lines as Unicode whitespace and
document the tokenizer overrides
discard b5723b45d OPENNLP-1894: Apply the review-convention pass and drop
unreferenced lexicon accessors
discard 104cd0e29 OPENNLP-1894: Document lattice CJK tokenization with a
mirror-tested example
discard 73d8aa53d OPENNLP-1894: Chain lattice nodes intrusively instead of
allocating per-position lists
discard ae38eb423 OPENNLP-1894: Hold the lexicon in a double-array trie with
frequency-recoded labels
discard b48a8eb38 OPENNLP-1894: Precompute category runs, unbox the trie, and
reject inexpressible dictionary values
discard 4d76e948d OPENNLP-1894: Categorize by code point, keep unknown
candidates inside their category run, and validate context ids at load
discard 759af0e64 OPENNLP-1894: Document dictionary acquisition with a
checksum-verifying download helper
discard 040ea88dd OPENNLP-1894: Keep lattice test sources ASCII-only via
Unicode escapes, add an EUC-JP loading example
discard 8730659ac OPENNLP-1894: Usage example and edge-case tests for the
lattice and unigram segmenters, corrected javadoc
discard e29862206 OPENNLP-1894: Frequency-driven segmentation over
user-supplied lexicons
discard 1132c127d OPENNLP-1894: Character trie for lattice prefix search
discard 3ce137626 OPENNLP-1894: Lattice segmentation over user-supplied
mecab-format dictionaries
add 9951bfd0a OPENNLP-1870: Add opt-in emoji annotations for NameFinder,
Doccat, and Sentiment (#1177)
add 8eef045b6 Minor: Regenerated NOTICE File for
9951bfd0ae0aee8ca77eb436702e5544fbea2afc
add 66526b528 [maven-release-plugin] prepare release opennlp-3.0.0-M5
add a8642301f [maven-release-plugin] prepare for next development iteration
add 497e7af00 OPENNLP-1894: Lattice segmentation over user-supplied
mecab-format dictionaries
add cfc938272 OPENNLP-1894: Character trie for lattice prefix search
add 6e72cfc58 OPENNLP-1894: Frequency-driven segmentation over
user-supplied lexicons
add a43c3a73c OPENNLP-1894: Usage example and edge-case tests for the
lattice and unigram segmenters, corrected javadoc
add 88240817e OPENNLP-1894: Keep lattice test sources ASCII-only via
Unicode escapes, add an EUC-JP loading example
add 52cdea4d7 OPENNLP-1894: Document dictionary acquisition with a
checksum-verifying download helper
add c895aa51f OPENNLP-1894: Categorize by code point, keep unknown
candidates inside their category run, and validate context ids at load
add 6e0ec1f11 OPENNLP-1894: Precompute category runs, unbox the trie, and
reject inexpressible dictionary values
add 2d1680b73 OPENNLP-1894: Hold the lexicon in a double-array trie with
frequency-recoded labels
add 80f97253e OPENNLP-1894: Chain lattice nodes intrusively instead of
allocating per-position lists
add 3741068de OPENNLP-1894: Document lattice CJK tokenization with a
mirror-tested example
add 80e9229d6 OPENNLP-1894: Apply the review-convention pass and drop
unreferenced lexicon accessors
add db5f6cc06 OPENNLP-1894: Trim parsed lines as Unicode whitespace and
document the tokenizer overrides
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (65579e9cd)
\
N -- N -- N refs/heads/OPENNLP-1894-lattice-cjk (db5f6cc06)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
LICENSE | 8 +-
NOTICE | 9 +
.../tools/doccat/EmojiFeatureGenerator.java | 82 +++++++
.../sentiment/EmojiSentimentContextGenerator.java | 87 +++++++
.../tools/sentiment/EmojiSentimentFactory.java} | 46 ++--
.../EmojiAnnotationFeatureGenerator.java | 83 +++++++
...=> EmojiAnnotationFeatureGeneratorFactory.java} | 24 +-
.../tools/util/normalizer/EmojiAnnotation.java | 197 ++++++++++++++++
.../tools/util/normalizer/EmojiAnnotationJoin.java | 51 ++++
.../tools/util/normalizer/EmojiAnnotations.java | 256 +++++++++++++++++++++
.../tools/util/normalizer/EmojiAnnotator.java | 197 ++++++++++++++++
.../tools/util/normalizer/EmojiCategory.java | 62 +++++
.../tools/util/normalizer/EmojiEntityType.java | 57 +++++
.../opennlp/tools/util/normalizer/EmojiFlags.java | 227 ++++++++++++++++++
.../tools/util/normalizer/emoji-annotations.txt | 212 +++++++++++++++++
.../tools/doccat/EmojiFeatureGeneratorTest.java | 80 +++++++
.../EmojiSentimentContextGeneratorTest.java | 72 ++++++
.../EmojiAnnotationFeatureGeneratorTest.java | 127 ++++++++++
.../util/normalizer/EmojiAnnotationsTest.java | 256 +++++++++++++++++++++
.../tools/util/normalizer/EmojiAnnotatorTest.java | 176 ++++++++++++++
.../normalizer/EmojiAnnotatorUsageExampleTest.java | 65 ++++++
.../tools/util/normalizer/EmojiFlagsTest.java | 147 ++++++++++++
.../tools/util/normalizer/NormalizerTestUtil.java | 29 ++-
opennlp-distr/src/main/readme/LICENSE | 5 +-
opennlp-distr/src/main/readme/NOTICE | 9 +
opennlp-docs/src/docbkx/normalizer.xml | 67 +++++-
src/license/NOTICE.template | 9 +
27 files changed, 2586 insertions(+), 54 deletions(-)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
copy
opennlp-core/{opennlp-ml/opennlp-ml-commons/src/main/java/opennlp/tools/ml/model/ObjectDataReader.java
=>
opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java}
(50%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
copy
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/{CharacterNgramFeatureGeneratorFactory.java
=> EmojiAnnotationFeatureGeneratorFactory.java} (60%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/doccat/EmojiFeatureGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentiment/EmojiSentimentContextGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorUsageExampleTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiFlagsTest.java
copy
opennlp-api/src/main/java/opennlp/tools/util/normalizer/CharSequenceNormalizer.java
=>
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NormalizerTestUtil.java
(57%)
