This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1850-4-docs
in repository https://gitbox.apache.org/repos/asf/opennlp.git
discard 665a35ca9 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
discard a976b1807 OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
discard 9efc8dcb6 OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
discard ab2413f3e OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
discard 28e7d941e OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
discard 8437e4293 OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
discard f174dc8b9 OPENNLP-1850 Tighten normalizer manual wording (review nits)
discard 503c7a7e4 OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
discard 443b41886 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
discard 59ec57b99 OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
discard 8c6c1d65e OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
discard 3cee4aee7 OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
discard 018ce5835 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
discard e15875b0a OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
discard e81f23877 OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
discard f7363f385 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
discard f4832b714 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
discard d4317a4eb OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
discard 0e2d54911 OPENNLP-1850 Fail loud on corrupt document-classification
model output
discard 993dce689 OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
discard a7a419991 OPENNLP-1850 Harden fail-loud paths in the DL components
discard ddfaed50d OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
discard ba1668092 OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
discard 5bf589a14 OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
discard efe5506f7 OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
discard f84ba8cc1 OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
discard ec916b649 OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
discard 9646cd145 OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
discard 59329f1ed OPENNLP-1850 Per-language NormalizationProfile registry (2c)
add fefa03c22 OPENNLP-1850 Review: thread-safe Term cache, argument
validation, drop 'canonical' framing
add 827bf8e52 OPENNLP-1850 Per-language NormalizationProfile registry (2c)
add cc10d1a8c OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
add c7934bd9a OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
add cfc3de91d OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
add 19ae614df OPENNLP-1850 Review: validate NormalizationProfile components
add 2ec0365cf OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
add 1f8e1f831 OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
add f572435cb OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
add 61cf124da OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
add 4186a7b39 OPENNLP-1850 Harden fail-loud paths in the DL components
add 6db5d1d1a OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
add dc75d9e38 OPENNLP-1850 Fail loud on corrupt document-classification
model output
add d0f8a59b7 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
add bcca50498 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
add 2dd1783c8 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
add b92141f0d OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
add c33166f32 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
add 237932518 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
add c73f03a77 OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
add e50b6ee28 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
add 2dccba47b OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
add f622cd5a2 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
add 05813be23 OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
add 6a6c3bf95 OPENNLP-1850 Tighten normalizer manual wording (review nits)
add 917afbecb OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
add e0bdfd8e0 OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
add ca68ccec9 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
add 6a9b2ef4d OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
add 0f71888e5 OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
add 15ecb62e1 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (665a35ca9)
\
N -- N -- N refs/heads/OPENNLP-1850-4-docs (15ecb62e1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../opennlp/tools/util/normalizer/Dimension.java | 4 +-
.../util/normalizer/NormalizationProfile.java | 21 ++-
.../java/opennlp/tools/util/normalizer/Term.java | 56 +++++---
.../tools/util/normalizer/TermAnalyzer.java | 82 +++++++++---
.../util/normalizer/NormalizationProfilesTest.java | 10 ++
.../tools/util/normalizer/TermAnalyzerTest.java | 142 ++++++++++++++++++++-
6 files changed, 274 insertions(+), 41 deletions(-)
