[
https://issues.apache.org/jira/browse/OPENNLP-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096643#comment-18096643
]
Kristian Rickert commented on OPENNLP-1880:
-------------------------------------------
PoC successful. Will start when dependent tickets are closed
> Lexical-semantic layer
> ----------------------
>
> Key: OPENNLP-1880
> URL: https://issues.apache.org/jira/browse/OPENNLP-1880
> Project: OpenNLP
> Issue Type: New Feature
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h2. Summary
> OpenNLP has no lexical-semantic layer: no synonym lookup, no is-a hierarchy,
> and its lemmatizers are either trained models or flat dictionaries. This
> ticket adds the lexical knowledge base seam and two clean-room readers for
> the WordNet family of resources. The contracts land in opennlp-api (package
> {{opennlp.tools.wordnet}}): {{LexicalKnowledgeBase}} (lemma and synset lookup
> with typed relation navigation), {{Synset}}, {{WordNetPos}}, and
> {{WordNetRelation}}. The seam interface carries a generic name rather than a
> brand: a legacy Princeton database, an Open English WordNet release, a future
> bundled permissively-licensed lexicon, and a future user-downloaded lexicon
> are all implementations behind the one contract, and synset identity is
> opaque and source-qualified.
> A new {{opennlp-extensions/opennlp-wordnet}} module ships two readers
> producing equivalent immutable, thread-safe lexicon views: a StAX reader for
> WN-LMF XML (the Global WordNet Association interchange format used by Open
> English WordNet and many other language wordnets) and a reader for legacy
> Princeton WNDB directories. Both are clean-room, built from the published
> format documentation; no third-party WordNet library is used or referenced.
> The XML reader is hardened against XXE per the OWASP posture for
> DOCTYPE-bearing formats: the DOCTYPE declaration real releases ship is
> tokenized and skipped while external entities and the external DTD subset
> stay disabled, so an unmodified Open English WordNet download parses
> directly, and a dedicated test proves a DOCTYPE-declared internal-subset
> entity payload fails loud rather than expanding. On top of the seam,
> {{MorphyLemmatizer}} implements the existing {{Lemmatizer}} interface with
> the documented Morphy algorithm: exception-list lookup first, then
> per-part-of-speech detachment rules with every candidate validated against
> the lexicon.
> The module ships code only; users point it at a WordNet database they
> downloaded. Verified against Princeton WordNet 3.0 (117,659 synsets) and Open
> English WordNet 2024 (120,630 synsets).
> h2. Scope
> * Contracts in opennlp-api: {{LexicalKnowledgeBase}}, {{Synset}},
> {{WordNetPos}}, {{WordNetRelation}} (28 relation types plus the derived
> aliases the formats express).
> * {{WnLmfReader}}: XXE-hardened StAX reader for WN-LMF XML; sense relations
> lifted to the synset level; fail-loud structural validation naming the
> resource and line.
> * {{WndbReader}}: reader for Princeton WNDB directories (index and data files
> for all four parts of speech); offset-contract validation; adjective
> satellite normalization and syntactic-marker stripping.
> * A structural equivalence test over matching miniature WN-LMF and WNDB
> fixtures, so the two readers are pinned to produce the same lexicon.
> * {{MorphyLemmatizer}} plus {{MorphyExceptions}} (the {{*.exc}}
> exception-list reader), implementing the existing {{Lemmatizer}} interface.
> h2. Acceptance criteria
> * Both readers load their real full-size counterparts (Princeton 3.0 and OEWN
> 2024) and agree structurally on the shared miniature fixtures.
> * The XXE posture is tested: an unmodified DOCTYPE-bearing document parses;
> an internal-subset entity payload fails loud without expansion.
> * All lexicon views are immutable and thread-safe, with a concurrency test.
> * No lexicon data in the source tree or artifacts; no new dependencies; no
> third-party WordNet code consulted or referenced.
> h2. Out of scope
> * Bundling any lexicon data, including the exception lists (rationale
> documented in the javadoc; a bundled permissively-licensed lexicon is a
> later, separately-licensed decision).
> * Sense keys and {{index.sense}} (a later sense-inventory layer; the v1
> contract has no sense-key surface).
> * Similarity measures and query expansion features that stack on the seam.
> * The {{DictionaryLemmatizer}} javadoc discrepancy (documents "0" but returns
> "O"), a separate one-line fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
