[
https://issues.apache.org/jira/browse/OPENNLP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1841:
------------------------------------
Fix Version/s: (was: 3.0.0)
> Provide SymSpell spell-correction dictionary models
> (opennlp-models-spellcheck-{lang})
> --------------------------------------------------------------------------------------
>
> Key: OPENNLP-1841
> URL: https://issues.apache.org/jira/browse/OPENNLP-1841
> Project: OpenNLP
> Issue Type: New Feature
> Reporter: Richard Zowalla
> Priority: Major
> Fix For: 3.0.0-M4
>
>
> OPENNLP-1832 landed the engine and API (opennlp-spellcheck module:
> SpellChecker, the SymSpell engine, SymSpellModel, serializer, model-resolver,
> CLI). What it deliberately did not ship is data: the module contains no
> production dictionaries, and SymSpellModelResolver resolves nothing on a
> stock classpath.
> This issue covers producing, licensing, and publishing the actual dictionary
> model artifacts - one Maven jar per language, named
> opennlp-models-spellcheck-{lang} — so that new
> SymSpellModelResolver().resolveByLanguage("en") returns a usable model out of
> the box, consistent with how every other OpenNLP component ships pre-trained
> models.
> Proposed work
> 1. Decide the home for the artifacts
>
> I would recommend to produce them in apache/opennlp-models (same place other
> pre-trained models are released and where opennlp-models-* GAV coordinates
> live
> 2. Source MIT-/ASLv2-compatible frequency data
> For each target language, obtain a permissively-licensed unigram (and where
> available bigram) frequency list:
> - English: SymSpell reference frequency_dictionary_en_82_765.txt +
> frequency_bigramdictionary_en_243_342.txt (MIT) - gives us
> compound-correction (lookupCompound) coverage immediately.
> - Other languages: Hermit Dave's FrequencyWords (OpenSubtitles-derived,
> MIT) and/or Wortschatz/Leipzig corpora and/or Wikipedia Snapshots - license
> must be confirmed per language and recorded.
> - Every source's license, URL, and retrieval date go into NOTICE / LICENSE
> and the per-jar provenance, same discipline as OPENNLP-1832's data-licensing
> work.
> 3. Initial language set (proposal)
> en (with bigrams) first as the reference; then de, es, fr, nl, it as a
> second wave, gated on confirmed licensing.
> For each language, run SpellCheckModelBuilder (default maxEditDistance=2,
> prefixLength=7) and package via writePackage(...) so each jar contains *.bin
> + model.properties. Pin and record:
> - model.version (start at 1.0),
> - model.sha256 (computed by SymSpellModels over the binary),
> - the build parameters and source-data version, for reproducibility.
> Can also be scripted like it is done for the other models and data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
