It does - but not only embeddings - OpenNLP needs SentencePiece to run SentencePiece-backed neural annotators over sentences - like our own onnx features - while preserving exact offsets into the original text. This is the main reason I wrote it. All analysis, even in Python packages in this area does not handle this well (if at all), making it hard to create truly accurate highlighters without it. However, that analysis and offset tracking belongs in a core API layer. So it's not adding a dependency; it's just an interface that tracks the offsets for it and perhaps some associated metadata at that level.
I'll take a second look at SentencePiece and carve out the API/core stuff; the rest goes into addon. That's the reasoning: embeddings and SentencePiece are already central to much NLP work, so the interface can exist there. As new technologies emerge to fill this gap, we can pivot (like how the gRPC server uses OpenVINO, TEI, or Model2Vec, for example). Thoughts? On Tue, Sep 1, 2026, 9:02 AM Richard Zowalla <[email protected]> wrote: > Hss todo with embeddings, no? > That would contract in Martin mail because he wrote "PRs around > "embeddings" should for now live and be maintained in opennlp-addons.“. > > Gruß > Richard > > > Am 01.09.2026 um 14:33 schrieb Kristian Rickert <[email protected]>: > > > > Quick clarification: > > > > In the core/addons thread, Martin wrote that OPENNLP-1885, OPENNLP-1893, > > OPENNLP-1909, and possibly OPENNLP-1894 should fit in core, and that he > > wanted to put effort into them so M6 could mature. > > > > The M6 proposal includes OPENNLP-1893, OPENNLP-1909, and OPENNLP-1894, > but > > not OPENNLP-1885. Since the two messages were posted about a minute > apart, > > I suspect they may have crossed? > > > > > > > > > > > > On Tue, Sep 1, 2026 at 3:12 AM Richard Zowalla <[email protected]> wrote: > > > >> Subject: [DISCUSS] 3.0.0-M6 release > >> > >> Hi all, > >> > >> M5 went out on 2026-07-21, so we're about six weeks out and main has > >> accumulated ~30 commits since. I'd like to cut 3.0.0-M6 rather than let > >> that pile grow further and we have a target around 08/18. > >> > >> What's already on main since M5: > >> > >> - OPENNLP-1888 Document annotation container with typed, offset-anchored > >> layers (#1182). This is the biggest one and the one several other PRs > >> build on, so getting it into users' hands early matters. > >> - OPENNLP-1903 BeamSearch: chain nodes instead of per-candidate Sequence > >> copies (#1205) > >> - OPENNLP-1906 sentence detector abbreviation veto was quadratic in > >> document length (#1208) > >> - OPENNLP-1904 / -1905 default-locale case folding breaking Morfologik > >> lookup and lemmatizer models (#1206, #1207) > >> - OPENNLP-1922 ReDoS / stack exhaustion in the RegexNameFinderFactory > >> EMAIL and URL patterns (#1250) > >> - OPENNLP-1899 hardened SymSpell deserialization (#1197) > >> - OPENNLP-1902 cached models verified against their SHA-512 checksum > >> (#1239) > >> - OPENNLP-1900 documented and published security model (#1198) > >> - OPENNLP-1913 / -1915 Maven Wrapper in the source release, mvnw.cmd > >> line endings (#1229, #1231) > >> > >> That's already a reasonable milestone on its own. On top of that I'd > >> propose we try to land these four before we cut: > >> > >> 1. OPENNLP-1921 Paragraph normalizer for Gutenberg-style and Markdown > >> text (#1249, tusharsaini18899). Approved already, small, and it slots > >> into the normalizer chain we cleaned up in OPENNLP-1916. > >> > >> 2. OPENNLP-1893 Hunspell affix dictionaries for stemming (#1190, > >> krickert). Classified as core in the scope thread without objection: > >> fits the existing abstractions, no new dependencies, no bundled data. > >> Currently "changes requested" - if that round gets addressed I'd like > >> it in M6. (#1191, krickert). > >> > >> 3. OPENNLP-1909 Verified installer for user-supplied third-party > >> resources (#1211, krickert). Interface in core, > >> concrete downloaders in the modules they need it - M6 is a good place > >> to > >> try that split out in practice rather than on the list. > >> > >> 4. OPENNLP-1894 Dictionary-based tokenization for Japanese, Korean and > >> Chinese (#1191, krickert); open up for Asian language feels natural in > >> a globalized world. > >> > >> To avoid a misunderstanding: putting something > >> into a milestone is not the same as deciding it belongs in core for > >> 3.0.0 final. Milestones are explicitly not API-frozen - that is what the > >> M is for. If CJK tokenization or the installer ships in M6 and we later > >> agree it belongs in addons, moving it before the final release is > >> exactly the kind of change a milestone series is supposed to absorb. I'd > >> rather we get these in front of users and collect feedback than hold a > >> release while the scope thread converges. > >> > >> So M6 does not preempt > >> > >> https://lists.apache.org/thread/vc8mnvfp6rmr5qb2s9y6hw0zpcxm6t68 > >> > >> and I'd like that thread to keep running in parallel. The open questions > >> there - the review bar for addons, the compatibility contract, who > >> drives discoverability - are release-cadence questions for addons, not > >> or a core milestone. Same for the addons cleanup and getting > >> it to Maven Central: worth doing, independent of M6. > >> > >> If anything on the list above isn't ready, it drops from M6 and goes > >> into M7 (or GA???). I'd rather cut on a date than on a feature list. > >> > >> Objections, additions, or things I've missed that should be in M6? > >> > >> Gruß > >> Richard > >
