This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1850-2a-tokenizer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 4dbd6a2e9 OPENNLP-1850 Perf: hoist the per-char volatile reads in
WordBreakProperty/ExtendedPictographic
omit 80c83e8f5 OPENNLP-1850 Review: drop lazy-init justification comments
in WordBreakProperty/ExtendedPictographic
omit c53e3c650 OPENNLP-1850 Review nits: ExtendedPictographic fail-loud
parity + doc; WordType heuristic note (tokenizer)
omit 8ed849eb7 OPENNLP-1850 Fail loud on a Word_Break line missing its ';'
(tokenizer)
omit 5857d8717 OPENNLP-1850 UAX #29 word tokenizer: WordSegmenter,
WordTokenizer, WordType (2a)
omit c73ebeb7d OPENNLP-1850 Review: make AlignedText.normalized a
CharSequence; add normalizedString()
omit 051c6cd84 OPENNLP-1850 Review nits: soften DL forward-link; fix
LineBreakPreserving opener (alignment)
omit e12816043 OPENNLP-1850 Offset/alignment layer: Alignment, AlignedText,
buildAligned, *Aligned (1b)
add 2f53530e1 OPENNLP-1850 Add Unicode License V3 to the binary
distribution LICENSE
add 9037a09f6 OPENNLP-1850 Review: rename ConfusablesLoadTest to
ConfusablesParseTest
add 4ba1ac300 OPENNLP-1850 Offset/alignment layer: Alignment, AlignedText,
buildAligned, *Aligned (1b)
add 0d0a07e47 OPENNLP-1850 Review nits: soften DL forward-link; fix
LineBreakPreserving opener (alignment)
add 1bf057ce3 OPENNLP-1850 Review: make AlignedText.normalized a
CharSequence; add normalizedString()
add ba7e0cded OPENNLP-1850 Review: IAE null contract, @ThreadSafe, UID
regeneration, line-break rung test
add 846cd534c OPENNLP-1850 UAX #29 word tokenizer: WordSegmenter,
WordTokenizer, WordType (2a)
add 1ae24758a OPENNLP-1850 Fail loud on a Word_Break line missing its ';'
(tokenizer)
add 1a4274a75 OPENNLP-1850 Review nits: ExtendedPictographic fail-loud
parity + doc; WordType heuristic note (tokenizer)
add 64383f28b OPENNLP-1850 Review: drop lazy-init justification comments
in WordBreakProperty/ExtendedPictographic
add 0ab9e5b2e OPENNLP-1850 Perf: hoist the per-char volatile reads in
WordBreakProperty/ExtendedPictographic
add 393b4daca OPENNLP-1850 Review: uax29 javadoc pass, IAE guards on
WordTokenizer, loader consistency, new tests
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4dbd6a2e9)
\
N -- N -- N refs/heads/OPENNLP-1850-2a-tokenizer (393b4daca)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../opennlp/tools/util/normalizer/Alignment.java | 5 ++
.../opennlp/tools/util/normalizer/CharClass.java | 71 +++++++++++-----
.../tools/util/normalizer/CharClassTest.java | 31 +++++++
.../tools/tokenize/uax29/ExtendedPictographic.java | 39 +++++++--
.../tools/tokenize/uax29/WordBreakProperty.java | 85 +++++++++++++++----
.../tools/tokenize/uax29/WordSegmenter.java | 2 +-
.../tools/tokenize/uax29/WordTokenizer.java | 62 ++++++++++----
.../opennlp/tools/tokenize/uax29/WordType.java | 21 +++--
.../AlignedAggregateCharSequenceNormalizer.java | 2 +-
.../normalizer/BulletCharSequenceNormalizer.java | 2 +-
.../normalizer/DashCharSequenceNormalizer.java | 2 +-
.../normalizer/DigitCharSequenceNormalizer.java | 2 +-
.../normalizer/EllipsisCharSequenceNormalizer.java | 2 +-
.../GermanUmlautCharSequenceNormalizer.java | 2 +-
.../InvisibleCharSequenceNormalizer.java | 2 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 2 +-
.../normalizer/QuoteCharSequenceNormalizer.java | 2 +-
.../WhitespaceCharSequenceNormalizer.java | 2 +-
.../uax29/WordBoundaryConformanceTest.java | 28 +++++--
.../tokenize/uax29/WordBreakPropertyTest.java | 17 +++-
.../tools/tokenize/uax29/WordBreakTest.java | 75 +++++++++++++++++
.../tools/tokenize/uax29/WordTokenizerTest.java | 11 +++
.../opennlp/tools/tokenize/uax29/WordTypeTest.java | 93 +++++++++++++++++++++
...blesLoadTest.java => ConfusablesParseTest.java} | 2 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 97 ++++++++++++++++++++++
opennlp-distr/src/main/readme/LICENSE | 47 ++++++++++-
26 files changed, 619 insertions(+), 87 deletions(-)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/uax29/WordBreakTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/uax29/WordTypeTest.java
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/{ConfusablesLoadTest.java
=> ConfusablesParseTest.java} (98%)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizerTest.java
