This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1868
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 458e40709 OPENNLP-1868: Bundled full case-folding rung with provenance
and audit
omit dc4df3cbb OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
omit ea971822a OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
omit c5c58d463 OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
omit 14a2721be OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
omit ed6ed0384 OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
omit 08c58b41b OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
omit b46138824 OPENNLP-1850 Tighten normalizer manual wording (review nits)
omit dd95cb8fc OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
omit ebfa852f4 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
omit 0a88e363f OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
omit 3a813f33c OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
omit dac1ddf22 OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
omit 125162bd8 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
omit 4515f61dd OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
omit 9aec5a624 OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
omit 9b783fe06 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
omit 97d86eb19 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
omit fa87d9887 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
omit 6f2dd62a0 OPENNLP-1850 Fail loud on corrupt document-classification
model output
omit 200d05219 OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
omit b213e99ba OPENNLP-1850 Harden fail-loud paths in the DL components
omit 35e8e64b9 OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
omit d28988c47 OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
omit 93c141ad3 OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
omit 9e89e779d OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
omit 99d108c6c OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
omit 4d67305f4 OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
omit 32eac82de OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
omit a6ed01e4d OPENNLP-1850 Per-language NormalizationProfile registry (2c)
omit 1b1f99fcd OPENNLP-1850 Term model hardening: fail loud on a
contract-violating Lemmatizer
omit df04c058b OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
omit 214c94253 OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
omit cad0e810b OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
omit de75f594f OPENNLP-1850 Review: uax29 javadoc pass, IAE guards on
WordTokenizer, loader consistency, new tests
omit a62439329 OPENNLP-1850 Perf: hoist the per-char volatile reads in
WordBreakProperty/ExtendedPictographic
omit 721f08b79 OPENNLP-1850 Review: drop lazy-init justification comments
in WordBreakProperty/ExtendedPictographic
omit a44de61c7 OPENNLP-1850 Review nits: ExtendedPictographic fail-loud
parity + doc; WordType heuristic note (tokenizer)
omit 11ea9367e OPENNLP-1850 Fail loud on a Word_Break line missing its ';'
(tokenizer)
omit 3a993d160 OPENNLP-1850 UAX #29 word tokenizer: WordSegmenter,
WordTokenizer, WordType (2a)
add 47fd462a0 OPENNLP-1859: Add tests for BilouCodec encode/decode and
outcome compatibility (#1135)
add e2ffecd8a OPENNLP-1862: UAX #29 word tokenizer — WordSegmenter,
WordTokenizer, WordType (#1110)
add 2260b55ec Minor: Regenerated NOTICE File for
e2ffecd8a278653969d39045dff4dccfbdc9569c (#1142)
add e67c8bd29 OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
add 69906de04 OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
add 1cb0bb03a OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
add 37ce9e128 OPENNLP-1850 Term model hardening: fail loud on a
contract-violating Lemmatizer
add fefa03c22 OPENNLP-1850 Review: thread-safe Term cache, argument
validation, drop 'canonical' framing
add 827bf8e52 OPENNLP-1850 Per-language NormalizationProfile registry (2c)
add cc10d1a8c OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
add c7934bd9a OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
add cfc3de91d OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
add 19ae614df OPENNLP-1850 Review: validate NormalizationProfile components
add 2ec0365cf OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
add 1f8e1f831 OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
add f572435cb OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
add 61cf124da OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
add 4186a7b39 OPENNLP-1850 Harden fail-loud paths in the DL components
add 6db5d1d1a OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
add dc75d9e38 OPENNLP-1850 Fail loud on corrupt document-classification
model output
add d0f8a59b7 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
add bcca50498 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
add 2dd1783c8 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
add b92141f0d OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
add c33166f32 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
add 237932518 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
add 659f33ddc OPENNLP-1850 DL review nits: document arg-validation
contract, javadoc locate(), hoist span-ordering constant
add f80b1b58b OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
add cdd147300 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
add 3a229732c OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
add ef552ef73 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
add f3d8c6bb2 OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
add 1de3995b4 OPENNLP-1850 Tighten normalizer manual wording (review nits)
add c12d0f793 OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
add 3a35c124b OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
add 1b899aae6 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
add efbc88590 OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
add 4f2fc0b28 OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
add 130b5ed48 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
add 87f7bde6a OPENNLP-1868: Bundled full case-folding rung with provenance
and audit
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (458e40709)
\
N -- N -- N refs/heads/OPENNLP-1868 (87f7bde6a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../tools/namefind/OffsetMappingNameFinder.java | 2 +
.../java/opennlp/dl/namefinder/NameFinderDL.java | 31 +-
.../opennlp/tools/util/normalizer/Dimension.java | 4 +-
.../util/normalizer/NormalizationProfile.java | 21 +-
.../java/opennlp/tools/util/normalizer/Term.java | 56 ++-
.../tools/util/normalizer/TermAnalyzer.java | 82 ++++-
.../opennlp/tools/namefind/BilouCodecTest.java | 381 +++++++++++++++++++++
.../util/normalizer/NormalizationProfilesTest.java | 10 +
.../tools/util/normalizer/TermAnalyzerTest.java | 142 +++++++-
opennlp-distr/src/main/readme/NOTICE | 28 +-
10 files changed, 698 insertions(+), 59 deletions(-)
