OpenNLP Devs, I've opened a batch of PRs aimed at 3.0. They're meant for async review, and none of them assume they will merge or land in 3.0 without the usual approvals. The point is to shape the API now, while it's still easy to change, so the features fit and there's room to add more in 3.x later. Feedback on the code or the API shape is welcome and encouraged.
Most of it shares one theme: normalization and tokenization that keep offsets, so a span in the output still maps back to the original text. That's the part the older normalizers couldn't give us. Ready for review: 1. Full Unicode case folding, offset-aware: https://github.com/apache/opennlp/pull/1138 2. Whitespace aligned to the Unicode White_Space property: https://github.com/apache/opennlp/pull/1150 3. Legacy CharSequenceNormalizers rewritten without regex, byte-identical output: https://github.com/apache/opennlp/pull/1151 4. Emoji and emoticon folding, both directions: https://github.com/apache/opennlp/pull/1164 5. Non-breaking performance follow-up on the normalization and tokenization hot paths: https://github.com/apache/opennlp/pull/1161 6. Thread-safe stemmers with a StemmerFactory and per-thread caching: https://github.com/apache/opennlp/pull/1163 7. Pure-Java SentencePiece inference with exact original-text spans: https://github.com/apache/opennlp/pull/1165 Still drafts, opened for design feedback (not proposed for merge yet): 8. Static embedding engine for distilled embedding tables: https://github.com/apache/opennlp/pull/1152 9. Gazetteer and geocoder API with a bundled Natural Earth dataset: https://github.com/apache/opennlp/pull/1154 10. Lexical knowledge base seam with WordNet readers and a Morphy lemmatizer: https://github.com/apache/opennlp/pull/1155 11. Light and minimal stemmer tiers: https://github.com/apache/opennlp/pull/1166 12. Lexical expansion, synonyms and hypernyms, over that knowledge base seam: https://github.com/apache/opennlp/pull/1167 If you've only got time for a few, the first four are the base upon which the rest build. Thanks, Kristian
