[
https://issues.apache.org/jira/browse/OPENNLP-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120469#comment-18120469
]
ASF GitHub Bot commented on OPENNLP-1927:
-----------------------------------------
krickert commented on PR #1270:
URL: https://github.com/apache/opennlp/pull/1270#issuecomment-5884087493
Eval build passed.
> Complete Hunspell dictionary support for stemming and analysis
> --------------------------------------------------------------
>
> Key: OPENNLP-1927
> URL: https://issues.apache.org/jira/browse/OPENNLP-1927
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
> {{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and
> {{KEEPCASE}}. Dictionaries can load successfully while producing different
> results from Hunspell. This behavior is documented and was noted in the
> [review of PR
> #1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
> Implement the remaining dictionary behavior needed for Hunspell-compatible
> stemming and morphological analysis. A valid dictionary using an unsupported
> feature is a compatibility gap, not malformed input. Users should not need to
> remove valid rules from their dictionaries.
> h3. Acceptance criteria
> * Audit the [Hunspell dictionary
> format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
> identify missing behavior affecting stemming and analysis.
> * Implement the six directives above and other identified gaps.
> * Default to rejecting unsupported directives that could affect results, and
> unknown directives whose effect cannot be established. Report the directive
> and source location. Recognized metadata that does not affect these
> operations should not cause rejection.
> * Allow partial compatibility only through an explicit option that reports
> skipped features. Continue rejecting malformed supported input in either mode.
> * Verify results against a pinned Hunspell implementation using focused
> fixtures and representative real dictionaries, respecting their licenses.
> * Document supported behavior, loading options, and remaining limitations.
> Spelling suggestions are separate functionality and are not part of this
> stemmer ticket.
> h3. Related work
> * [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
> Hunspell stemmer support. Keep this issue closed.
> * [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
> analysis corrections. Release-note tracking for that merged work is separate
> from the future compatibility work described here.
> h3. Release notes
> Hunspell dictionaries now load strictly by default:
> HunspellDictionary.load(...) throws an IOException for an unsupported affix
> directive. In 3.0.0-M6 unsupported directives were ignored. Pass
> HunspellDictionary.LoadMode.ALLOW_PARTIAL to keep the earlier behaviour and
> read the skipped directives through getUnsupportedDirectives().
> HunspellStemmer.analyze is package-private in 3.0.0; the public API returns
> stems.
> A typed analysis enhancement is detailed in github issue #1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
