[
https://issues.apache.org/jira/browse/OPENNLP-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120437#comment-18120437
]
ASF GitHub Bot commented on OPENNLP-1927:
-----------------------------------------
krickert commented on PR #1270:
URL: https://github.com/apache/opennlp/pull/1270#issuecomment-5881345277
@rzo1 @mawiesne @jzonthemtn , this is ready for another pass.
Merged as signed-off commit and an eval run:
https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/86/
AI summary, hand validated by me. I have the follow-ups (shared test helper
+ cleanup) completed that we'll make a tiny PR for after I get through these
tickets and some backport work. Future changes will be in an apache draft
(intent was to avoid noise on apache feeds):
> Since 3510fe6:
>
> - The fixtures that still reused Hunspell test material (zeroaffix, alias,
forbiddenword) are replaced with our own words and flags, outcomes recorded
from e184e22c. The recorded s tems are now listed per fixture in the test
instead of defaulting to the expectation.
> - A long token under a compound dictionary took cubic time in the
word-pair check. It is bounded by the longest spaced form the dictionary can
read.
> - A BOM hid a FLAG declaration on line 1; CR-only files failed to load;
NONGRAMSUGGEST and CHECKNUM were rejected under STRICT; an underscore inside an
ICONV, OCONV or REP pattern
> was not read as a space. All fixed, tests first.
> - Constants for the remaining repeated literals, "Thrown if" on every
@throws, and the review notes are out of the Javadoc.
>
> Not in this PR, tracked as follow-ups: the shared test fixture helper
across the four test classes, and a small internal cleanup of the case variant
scan.
>
>
Let me know if there's anything else. Hope you like ti.
> Complete Hunspell dictionary support for stemming and analysis
> --------------------------------------------------------------
>
> Key: OPENNLP-1927
> URL: https://issues.apache.org/jira/browse/OPENNLP-1927
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
> {{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and
> {{KEEPCASE}}. Dictionaries can load successfully while producing different
> results from Hunspell. This behavior is documented and was noted in the
> [review of PR
> #1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
> Implement the remaining dictionary behavior needed for Hunspell-compatible
> stemming and morphological analysis. A valid dictionary using an unsupported
> feature is a compatibility gap, not malformed input. Users should not need to
> remove valid rules from their dictionaries.
> h3. Acceptance criteria
> * Audit the [Hunspell dictionary
> format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
> identify missing behavior affecting stemming and analysis.
> * Implement the six directives above and other identified gaps.
> * Default to rejecting unsupported directives that could affect results, and
> unknown directives whose effect cannot be established. Report the directive
> and source location. Recognized metadata that does not affect these
> operations should not cause rejection.
> * Allow partial compatibility only through an explicit option that reports
> skipped features. Continue rejecting malformed supported input in either mode.
> * Verify results against a pinned Hunspell implementation using focused
> fixtures and representative real dictionaries, respecting their licenses.
> * Document supported behavior, loading options, and remaining limitations.
> Spelling suggestions are separate functionality and are not part of this
> stemmer ticket.
> h3. Related work
> * [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
> Hunspell stemmer support. Keep this issue closed.
> * [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
> analysis corrections. Release-note tracking for that merged work is separate
> from the future compatibility work described here.
> h3. Release notes
> Hunspell dictionaries now load strictly by default:
> HunspellDictionary.load(...) throws an IOException for an unsupported affix
> directive. In 3.0.0-M6 unsupported directives were ignored. Pass
> HunspellDictionary.LoadMode.ALLOW_PARTIAL to keep the earlier behaviour and
> read the skipped directives through getUnsupportedDirectives().
> HunspellStemmer.analyze is package-private in 3.0.0; the public API returns
> stems.
> A typed analysis enhancement is detailed in github issue #1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
