This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1894-lattice-cjk
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard b02948cfc OPENNLP-1894: Address review: complete javadoc, hoist
constants, and fold fixture duplication
discard db5f6cc06 OPENNLP-1894: Trim parsed lines as Unicode whitespace and
document the tokenizer overrides
discard 80e9229d6 OPENNLP-1894: Apply the review-convention pass and drop
unreferenced lexicon accessors
discard 3741068de OPENNLP-1894: Document lattice CJK tokenization with a
mirror-tested example
discard 80f97253e OPENNLP-1894: Chain lattice nodes intrusively instead of
allocating per-position lists
discard 2d1680b73 OPENNLP-1894: Hold the lexicon in a double-array trie with
frequency-recoded labels
discard 6e0ec1f11 OPENNLP-1894: Precompute category runs, unbox the trie, and
reject inexpressible dictionary values
discard c895aa51f OPENNLP-1894: Categorize by code point, keep unknown
candidates inside their category run, and validate context ids at load
discard 52cdea4d7 OPENNLP-1894: Document dictionary acquisition with a
checksum-verifying download helper
discard 88240817e OPENNLP-1894: Keep lattice test sources ASCII-only via
Unicode escapes, add an EUC-JP loading example
discard a43c3a73c OPENNLP-1894: Usage example and edge-case tests for the
lattice and unigram segmenters, corrected javadoc
discard 6e72cfc58 OPENNLP-1894: Frequency-driven segmentation over
user-supplied lexicons
discard cfc938272 OPENNLP-1894: Character trie for lattice prefix search
discard 497e7af00 OPENNLP-1894: Lattice segmentation over user-supplied
mecab-format dictionaries
add 71e1fd94f OPENNLP-1898: Address the OPENNLP-1876 review follow-ups
(#1196)
add b892c9d48 OPENNLP-1894: Lattice segmentation over user-supplied
mecab-format dictionaries
add 67d7643c6 OPENNLP-1894: Character trie for lattice prefix search
add 376e419b6 OPENNLP-1894: Frequency-driven segmentation over
user-supplied lexicons
add b96496662 OPENNLP-1894: Usage example and edge-case tests for the
lattice and unigram segmenters, corrected javadoc
add 264c5feab OPENNLP-1894: Keep lattice test sources ASCII-only via
Unicode escapes, add an EUC-JP loading example
add 41cb53573 OPENNLP-1894: Document dictionary acquisition with a
checksum-verifying download helper
add 7a4ee6e70 OPENNLP-1894: Categorize by code point, keep unknown
candidates inside their category run, and validate context ids at load
add d8870784b OPENNLP-1894: Precompute category runs, unbox the trie, and
reject inexpressible dictionary values
add 075b343c8 OPENNLP-1894: Hold the lexicon in a double-array trie with
frequency-recoded labels
add 19c58d265 OPENNLP-1894: Chain lattice nodes intrusively instead of
allocating per-position lists
add 7536cf13b OPENNLP-1894: Document lattice CJK tokenization with a
mirror-tested example
add e689f3f8d OPENNLP-1894: Apply the review-convention pass and drop
unreferenced lexicon accessors
add ceca48832 OPENNLP-1894: Trim parsed lines as Unicode whitespace and
document the tokenizer overrides
add 6716542d3 OPENNLP-1894: Address review: complete javadoc, hoist
constants, and fold fixture duplication
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (b02948cfc)
\
N -- N -- N refs/heads/OPENNLP-1894-lattice-cjk (6716542d3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
...CharSequenceNormalizerCharacterizationTest.java | 6 ++-
...CharSequenceNormalizerCharacterizationTest.java | 11 +++--
...CharSequenceNormalizerCharacterizationTest.java | 21 +++++----
...CharSequenceNormalizerCharacterizationTest.java | 54 +++++++++++++---------
.../SpellCheckingCharSequenceNormalizer.java | 9 +++-
.../stream/SpellCorrectingTokenStream.java | 8 ++--
6 files changed, 68 insertions(+), 41 deletions(-)
