This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1868
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard 952ff64f3 OPENNLP-1868: Bundled full case-folding rung with provenance
and audit
discard 914424ce8 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
discard ca3e7e20a OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
discard 6f2f456f1 OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
discard c03c60030 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
discard 248f7af20 OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
discard 090c5c96e OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
discard b909d1723 OPENNLP-1850 Tighten normalizer manual wording (review nits)
discard 7fa90d5c3 OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
discard 46d3d1dda OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
discard 54e06dcb5 OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
discard 72be8aeb7 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
discard 394a6ade1 OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
discard db3f5e812 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
discard 2374e9c65 OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
discard f0ffd5f5e OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
discard 6701e527b OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
discard 9d2e54d39 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
discard 6464d7319 OPENNLP-1850 Fail loud on corrupt document-classification
model output
discard 6b0283cd5 OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
discard 4455f866c OPENNLP-1850 Harden fail-loud paths in the DL components
discard daba0a5a7 OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
discard e703f024e OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
discard 1bcad3c9a OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
discard 1833b088b OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
discard ea80f2f83 OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
discard f18960131 OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
discard b7e574912 OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
discard 11efd78c7 OPENNLP-1850 Per-language NormalizationProfile registry (2c)
discard b0b2fa653 OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
discard 1ccc77263 OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
discard dfb687ac8 OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
discard 4dbd6a2e9 OPENNLP-1850 Perf: hoist the per-char volatile reads in
WordBreakProperty/ExtendedPictographic
discard 80c83e8f5 OPENNLP-1850 Review: drop lazy-init justification comments
in WordBreakProperty/ExtendedPictographic
discard c53e3c650 OPENNLP-1850 Review nits: ExtendedPictographic fail-loud
parity + doc; WordType heuristic note (tokenizer)
discard 8ed849eb7 OPENNLP-1850 Fail loud on a Word_Break line missing its ';'
(tokenizer)
discard 5857d8717 OPENNLP-1850 UAX #29 word tokenizer: WordSegmenter,
WordTokenizer, WordType (2a)
discard c73ebeb7d OPENNLP-1850 Review: make AlignedText.normalized a
CharSequence; add normalizedString()
discard 051c6cd84 OPENNLP-1850 Review nits: soften DL forward-link; fix
LineBreakPreserving opener (alignment)
discard e12816043 OPENNLP-1850 Offset/alignment layer: Alignment, AlignedText,
buildAligned, *Aligned (1b)
add 2f53530e1 OPENNLP-1850 Add Unicode License V3 to the binary
distribution LICENSE
add 9037a09f6 OPENNLP-1850 Review: rename ConfusablesLoadTest to
ConfusablesParseTest
add 4ba1ac300 OPENNLP-1850 Offset/alignment layer: Alignment, AlignedText,
buildAligned, *Aligned (1b)
add 0d0a07e47 OPENNLP-1850 Review nits: soften DL forward-link; fix
LineBreakPreserving opener (alignment)
add 1bf057ce3 OPENNLP-1850 Review: make AlignedText.normalized a
CharSequence; add normalizedString()
add ba7e0cded OPENNLP-1850 Review: IAE null contract, @ThreadSafe, UID
regeneration, line-break rung test
add 846cd534c OPENNLP-1850 UAX #29 word tokenizer: WordSegmenter,
WordTokenizer, WordType (2a)
add 1ae24758a OPENNLP-1850 Fail loud on a Word_Break line missing its ';'
(tokenizer)
add 1a4274a75 OPENNLP-1850 Review nits: ExtendedPictographic fail-loud
parity + doc; WordType heuristic note (tokenizer)
add 64383f28b OPENNLP-1850 Review: drop lazy-init justification comments
in WordBreakProperty/ExtendedPictographic
add 0ab9e5b2e OPENNLP-1850 Perf: hoist the per-char volatile reads in
WordBreakProperty/ExtendedPictographic
add 393b4daca OPENNLP-1850 Review: uax29 javadoc pass, IAE guards on
WordTokenizer, loader consistency, new tests
add 5270ac98c OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
add 6725eb99f OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
add e9400c2e7 OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
add 466c25c51 OPENNLP-1850 Per-language NormalizationProfile registry (2c)
add 4c5736b2f OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
add 97809e910 OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
add befb4cc70 OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
add ed2ab94ba OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
add 416d5d7be OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
add b19e4db52 OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
add cfb7a3b7f OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
add acf9e3878 OPENNLP-1850 Harden fail-loud paths in the DL components
add a48b7140d OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
add 90f3a0864 OPENNLP-1850 Fail loud on corrupt document-classification
model output
add 9bbc2cf96 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
add e6431d3c4 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
add b3c28680f OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
add 81abe3b81 OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
add 26280a0c1 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
add a5abd857f OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
add 321122023 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
add 0f6837d27 OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
add f69772dd4 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
add a8bbc687d OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
add 4916d7aed OPENNLP-1850 Tighten normalizer manual wording (review nits)
add 7492a7c0a OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
add 42eeea1de OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
add 4fb434776 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
add f7731d11b OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
add f3fc114ce OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
add 3aa333fa0 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
add c8fa95e53 OPENNLP-1868: Bundled full case-folding rung with provenance
and audit
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (952ff64f3)
\
N -- N -- N refs/heads/OPENNLP-1868 (c8fa95e53)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../opennlp/tools/util/normalizer/Alignment.java | 5 ++
.../opennlp/tools/util/normalizer/CharClass.java | 71 +++++++++++-----
.../tools/util/normalizer/CharClassTest.java | 31 +++++++
.../tools/tokenize/uax29/ExtendedPictographic.java | 39 +++++++--
.../tools/tokenize/uax29/WordBreakProperty.java | 85 +++++++++++++++----
.../tools/tokenize/uax29/WordSegmenter.java | 2 +-
.../tools/tokenize/uax29/WordTokenizer.java | 62 ++++++++++----
.../opennlp/tools/tokenize/uax29/WordType.java | 21 +++--
.../AlignedAggregateCharSequenceNormalizer.java | 2 +-
.../normalizer/BulletCharSequenceNormalizer.java | 2 +-
.../normalizer/DashCharSequenceNormalizer.java | 2 +-
.../normalizer/DigitCharSequenceNormalizer.java | 2 +-
.../normalizer/EllipsisCharSequenceNormalizer.java | 2 +-
.../GermanUmlautCharSequenceNormalizer.java | 2 +-
.../InvisibleCharSequenceNormalizer.java | 2 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 2 +-
.../normalizer/QuoteCharSequenceNormalizer.java | 2 +-
.../WhitespaceCharSequenceNormalizer.java | 2 +-
.../uax29/WordBoundaryConformanceTest.java | 28 +++++--
.../tokenize/uax29/WordBreakPropertyTest.java | 17 +++-
.../tools/tokenize/uax29/WordBreakTest.java | 75 +++++++++++++++++
.../tools/tokenize/uax29/WordTokenizerTest.java | 11 +++
.../opennlp/tools/tokenize/uax29/WordTypeTest.java | 93 +++++++++++++++++++++
...blesLoadTest.java => ConfusablesParseTest.java} | 2 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 97 ++++++++++++++++++++++
opennlp-distr/src/main/readme/LICENSE | 47 ++++++++++-
26 files changed, 619 insertions(+), 87 deletions(-)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/uax29/WordBreakTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/uax29/WordTypeTest.java
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/{ConfusablesLoadTest.java
=> ConfusablesParseTest.java} (98%)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizerTest.java
