This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1850-4-docs
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 130b5ed48 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
omit 4f2fc0b28 OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
omit efbc88590 OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
omit 1b899aae6 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
omit 3a35c124b OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
omit c12d0f793 OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
omit 1de3995b4 OPENNLP-1850 Tighten normalizer manual wording (review nits)
omit f3d8c6bb2 OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
omit ef552ef73 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
omit 3a229732c OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
omit cdd147300 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
omit f80b1b58b OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
omit 659f33ddc OPENNLP-1850 DL review nits: document arg-validation
contract, javadoc locate(), hoist span-ordering constant
omit 237932518 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
omit c33166f32 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
omit b92141f0d OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
omit 2dd1783c8 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
omit bcca50498 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
omit d0f8a59b7 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
omit dc75d9e38 OPENNLP-1850 Fail loud on corrupt document-classification
model output
omit 6db5d1d1a OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
omit 4186a7b39 OPENNLP-1850 Harden fail-loud paths in the DL components
omit 61cf124da OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
omit f572435cb OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
omit 1f8e1f831 OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
omit 2ec0365cf OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
omit 19ae614df OPENNLP-1850 Review: validate NormalizationProfile components
omit cfc3de91d OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
omit c7934bd9a OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
omit cc10d1a8c OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
omit 827bf8e52 OPENNLP-1850 Per-language NormalizationProfile registry (2c)
omit fefa03c22 OPENNLP-1850 Review: thread-safe Term cache, argument
validation, drop 'canonical' framing
omit 37ce9e128 OPENNLP-1850 Term model hardening: fail loud on a
contract-violating Lemmatizer
omit 1cb0bb03a OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
omit 69906de04 OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
omit e67c8bd29 OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
add 74092e1c9 OPENNLP-1873: Update ONNX runtime to 1.27.0 (#1143)
add 1712f32db OPENNLP-1874: Update Log4J to 2.26.1 (#1144)
add 945a3079f Minor: Regenerated NOTICE File for
1712f32dbe657792f9a80fa960018ab2a9eb2206 (#1147)
add 388585c02 OPENNLP-1850 Layered Term model: Term, TermAnalyzer (2b)
add 0ce3db3a4 OPENNLP-1850 Review nits: rename dashes()->dash(); LEMMA
doc+test; soften forward-link (Term)
add 3dbe1d33a OPENNLP-1850 Review nits: TermAnalyzer javadoc references
matchingAnalyzer()
add 0290ae56d OPENNLP-1850 Term model hardening: fail loud on a
contract-violating Lemmatizer
add bbfcd659b OPENNLP-1850 Review: thread-safe Term cache, argument
validation, drop 'canonical' framing
add 528073833 OPENNLP-1850 Review: IllegalArgumentException on null args;
TermTest + multilingual cases
add b1516f56e OPENNLP-1850 Per-language NormalizationProfile registry (2c)
add 066f3f0f6 OPENNLP-1850 Resolve Norwegian nb/nn to the Norwegian
profile (profiles)
add 01e3c07d1 OPENNLP-1850 Review nits: add Turkish profile; derive
coverage from the enum (profiles)
add 7cb5db52c OPENNLP-1850 Review nits: rename
searchAnalyzer->matchingAnalyzer; drop 'search' framing in profile docs
add f2232c787 OPENNLP-1850 Review: validate NormalizationProfile components
add f99fa6291 OPENNLP-1850 Review: IllegalArgumentException on null args;
link profile in javadoc
add feaf84296 OPENNLP-1850 Offset-safe, Unicode-aware input normalization
in the DL components
add 052740d1a OPENNLP-1850 Add OffsetMappingNameFinder capability
interface and a findInOriginal end-to-end test
add 1ee0d317c OPENNLP-1850 Resolve overlapping chunk spans and compose the
input alignment
add 0373eacbd OPENNLP-1850 Add real-model chunk-boundary eval tests; drop
dead label constants
add 6c15c1985 OPENNLP-1850 Harden fail-loud paths in the DL components
add e4c3f6d90 OPENNLP-1850 Fail fast on null finder input; fix the GPU
eval test options
add 4eece28fa OPENNLP-1850 Fail loud on corrupt document-classification
model output
add 0acd1f838 OPENNLP-1850 Fully-qualify TokenNameFinder javadoc links in
NameFinderDL
add a35d8d168 OPENNLP-1850 Reject non-finite logits in softmax, not just
NaN (dl)
add a7fef6166 OPENNLP-1850 Make mergeOverlappingSpans O(n log n) (dl)
add 28f875fbe OPENNLP-1850 Review nits: extract testable DL guards;
merge-copy; capitalize msgs; migration note
add 050895d61 OPENNLP-1850 Follow AlignedText.normalized() -> CharSequence
in NameFinderDL
add dd941c313 OPENNLP-1850 DL hardening: IllegalArgumentException null
contract and null-element guards
add 02b8917be OPENNLP-1850 DL review nits: document arg-validation
contract, javadoc locate(), hoist span-ordering constant
add 4eff9b957 OPENNLP-1850 Document Unicode normalization, the UAX #29
tokenizer, and DL handling
add 7e0b34797 OPENNLP-1850 Document the offset-aware normalization
pipeline (buildAligned)
add 8da862f65 OPENNLP-1850 Name the OffsetMappingNameFinder capability
interface in the manual
add 77b8c9627 OPENNLP-1850 Document the offset-aware substitution folds
(quotes, digits, ellipsis, bullets, umlaut)
add 342d9384d OPENNLP-1850 Document the supplementary-dash offset shift in
the DL fold options
add 102be3e6b OPENNLP-1850 Tighten normalizer manual wording (review nits)
add 3d753cb7c OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat
in the tokenizer manual
add 312a7aa03 OPENNLP-1850 Docs review nits: populate ids2Labels example;
rule-based hyphenation
add 058af2c01 OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate
second ids2Labels example
add 03f8ee54c OPENNLP-1850 Review nits: manual uses
defaultChain()/matchingAnalyzer(); drop BM25/search framing
add baf8dfc59 OPENNLP-1850 Align normalizer.xml DOCTYPE to the
OPENNLP-1854 local DTD catalog form
add 714cff029 OPENNLP-1850 Manual: qualify the dash-fold offset claim for
supplementary-plane dashes
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (130b5ed48)
\
N -- N -- N refs/heads/OPENNLP-1850-4-docs (714cff029)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
NOTICE | 10 +-
.../util/normalizer/NormalizationProfile.java | 14 +--
.../util/normalizer/NormalizationProfiles.java | 26 +++--
.../java/opennlp/tools/util/normalizer/Term.java | 18 ++--
.../tools/util/normalizer/TermAnalyzer.java | 76 +++++++++-----
.../util/normalizer/NormalizationProfilesTest.java | 28 +++++-
.../normalizer/TermAnalyzerMultilingualTest.java | 78 +++++++++++++++
.../tools/util/normalizer/TermAnalyzerTest.java | 26 ++---
.../opennlp/tools/util/normalizer/TermTest.java | 109 +++++++++++++++++++++
opennlp-distr/src/main/readme/NOTICE | 10 +-
pom.xml | 4 +-
11 files changed, 324 insertions(+), 75 deletions(-)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/TermAnalyzerMultilingualTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/TermTest.java
