Richard Zowalla created OPENNLP-1841:
----------------------------------------
Summary: Provide SymSpell spell-correction dictionary models
(opennlp-models-spellcheck-{lang})
Key: OPENNLP-1841
URL: https://issues.apache.org/jira/browse/OPENNLP-1841
Project: OpenNLP
Issue Type: New Feature
Reporter: Richard Zowalla
Fix For: 3.0.0-M4, 3.0.0
OPENNLP-1832 landed the engine and API (opennlp-spellcheck module:
SpellChecker, the SymSpell engine, SymSpellModel, serializer, model-resolver,
CLI). What it deliberately did not ship is data: the module contains no
production dictionaries, and SymSpellModelResolver resolves nothing on a stock
classpath.
This issue covers producing, licensing, and publishing the actual dictionary
model artifacts - one Maven jar per language, named
opennlp-models-spellcheck-{lang} — so that new
SymSpellModelResolver().resolveByLanguage("en") returns a usable model out of
the box, consistent with how every other OpenNLP component ships pre-trained
models.
Proposed work
1. Decide the home for the artifacts
I would recommend to produce them in apache/opennlp-models (same place other
pre-trained models are released and where opennlp-models-* GAV coordinates live
2. Source MIT-/ASLv2-compatible frequency data
For each target language, obtain a permissively-licensed unigram (and where
available bigram) frequency list:
- English: SymSpell reference frequency_dictionary_en_82_765.txt +
frequency_bigramdictionary_en_243_342.txt (MIT) - gives us compound-correction
(lookupCompound) coverage immediately.
- Other languages: Hermit Dave's FrequencyWords (OpenSubtitles-derived, MIT)
and/or Wortschatz/Leipzig corpora and/or Wikipedia Snapshots - license must be
confirmed per language and recorded.
- Every source's license, URL, and retrieval date go into NOTICE / LICENSE
and the per-jar provenance, same discipline as OPENNLP-1832's data-licensing
work.
3. Initial language set (proposal)
en (with bigrams) first as the reference; then de, es, fr, nl, it as a second
wave, gated on confirmed licensing.
For each language, run SpellCheckModelBuilder (default maxEditDistance=2,
prefixLength=7) and package via writePackage(...) so each jar contains *.bin +
model.properties. Pin and record:
- model.version (start at 1.0),
- model.sha256 (computed by SymSpellModels over the binary),
- the build parameters and source-data version, for reproducibility.
Can also be scripted like it is done for the other models and data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
