Kristian Rickert created OPENNLP-1880:
-----------------------------------------
Summary: Lexical-semantic layer
Key: OPENNLP-1880
URL: https://issues.apache.org/jira/browse/OPENNLP-1880
Project: OpenNLP
Issue Type: New Feature
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Fix For: 3.0.0-M5
h2. Summary
OpenNLP has no lexical-semantic layer: no synonym lookup, no is-a hierarchy,
and its lemmatizers are either trained models or flat dictionaries. This ticket
adds the lexical knowledge base seam and two clean-room readers for the WordNet
family of resources. The contracts land in opennlp-api (package
{{opennlp.tools.wordnet}}): {{LexicalKnowledgeBase}} (lemma and synset lookup
with typed relation navigation), {{Synset}}, {{WordNetPos}}, and
{{WordNetRelation}}. The seam interface carries a generic name rather than a
brand: a legacy Princeton database, an Open English WordNet release, a future
bundled permissively-licensed lexicon, and a future user-downloaded lexicon are
all implementations behind the one contract, and synset identity is opaque and
source-qualified.
A new {{opennlp-extensions/opennlp-wordnet}} module ships two readers producing
equivalent immutable, thread-safe lexicon views: a StAX reader for WN-LMF XML
(the Global WordNet Association interchange format used by Open English WordNet
and many other language wordnets) and a reader for legacy Princeton WNDB
directories. Both are clean-room, built from the published format
documentation; no third-party WordNet library is used or referenced. The XML
reader is hardened against XXE per the OWASP posture for DOCTYPE-bearing
formats: the DOCTYPE declaration real releases ship is tokenized and skipped
while external entities and the external DTD subset stay disabled, so an
unmodified Open English WordNet download parses directly, and a dedicated test
proves a DOCTYPE-declared internal-subset entity payload fails loud rather than
expanding. On top of the seam, {{MorphyLemmatizer}} implements the existing
{{Lemmatizer}} interface with the documented Morphy algorithm: exception-list
lookup first, then per-part-of-speech detachment rules with every candidate
validated against the lexicon.
The module ships code only; users point it at a WordNet database they
downloaded. Verified against Princeton WordNet 3.0 (117,659 synsets) and Open
English WordNet 2024 (120,630 synsets).
h2. Scope
* Contracts in opennlp-api: {{LexicalKnowledgeBase}}, {{Synset}},
{{WordNetPos}}, {{WordNetRelation}} (28 relation types plus the derived aliases
the formats express).
* {{WnLmfReader}}: XXE-hardened StAX reader for WN-LMF XML; sense relations
lifted to the synset level; fail-loud structural validation naming the resource
and line.
* {{WndbReader}}: reader for Princeton WNDB directories (index and data files
for all four parts of speech); offset-contract validation; adjective satellite
normalization and syntactic-marker stripping.
* A structural equivalence test over matching miniature WN-LMF and WNDB
fixtures, so the two readers are pinned to produce the same lexicon.
* {{MorphyLemmatizer}} plus {{MorphyExceptions}} (the {{*.exc}} exception-list
reader), implementing the existing {{Lemmatizer}} interface.
h2. Acceptance criteria
* Both readers load their real full-size counterparts (Princeton 3.0 and OEWN
2024) and agree structurally on the shared miniature fixtures.
* The XXE posture is tested: an unmodified DOCTYPE-bearing document parses; an
internal-subset entity payload fails loud without expansion.
* All lexicon views are immutable and thread-safe, with a concurrency test.
* No lexicon data in the source tree or artifacts; no new dependencies; no
third-party WordNet code consulted or referenced.
h2. Out of scope
* Bundling any lexicon data, including the exception lists (rationale
documented in the javadoc; a bundled permissively-licensed lexicon is a later,
separately-licensed decision).
* Sense keys and {{index.sense}} (a later sense-inventory layer; the v1
contract has no sense-key surface).
* Similarity measures and query expansion features that stack on the seam.
* The {{DictionaryLemmatizer}} javadoc discrepancy (documents "0" but returns
"O"), a separate one-line fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
