Hi Kristian, the clarification makes it clearer. The approach / direction outlined below sounds valid. Go for API/core carving.
Thx Martin > Am 02.09.2026 um 21:21 schrieb Kristian Rickert <[email protected]>: > > It does - but not only embeddings - > > OpenNLP needs SentencePiece to run SentencePiece-backed neural annotators > over sentences - like our own onnx features - while preserving exact > offsets into the original text. This is the main reason I wrote it. All > analysis, even in Python packages in this area does not handle this well > (if at all), making it hard to create truly accurate highlighters without > it. However, that analysis and offset tracking belongs in a core API > layer. So it's not adding a dependency; it's just an interface that tracks > the offsets for it and perhaps some associated metadata at that level. > > I'll take a second look at SentencePiece and carve out the API/core stuff; > the rest goes into addon. > > That's the reasoning: embeddings and SentencePiece are already central to > much NLP work, so the interface can exist there. As new technologies emerge > to fill this gap, we can pivot (like how the gRPC server uses OpenVINO, > TEI, or Model2Vec, for example). > > Thoughts? > > > On Tue, Sep 1, 2026, 9:02 AM Richard Zowalla <[email protected]> wrote: > >> Hss todo with embeddings, no? >> That would contract in Martin mail because he wrote "PRs around >> "embeddings" should for now live and be maintained in opennlp-addons.“. >> >> Gruß >> Richard >> >>> Am 01.09.2026 um 14:33 schrieb Kristian Rickert <[email protected]>: >>> >>> Quick clarification: >>> >>> In the core/addons thread, Martin wrote that OPENNLP-1885, OPENNLP-1893, >>> OPENNLP-1909, and possibly OPENNLP-1894 should fit in core, and that he >>> wanted to put effort into them so M6 could mature. >>> >>> The M6 proposal includes OPENNLP-1893, OPENNLP-1909, and OPENNLP-1894, >> but >>> not OPENNLP-1885. Since the two messages were posted about a minute >> apart, >>> I suspect they may have crossed? >>> >>> >>> >>> >>> >>> On Tue, Sep 1, 2026 at 3:12 AM Richard Zowalla <[email protected]> wrote: >>> >>>> Subject: [DISCUSS] 3.0.0-M6 release >>>> >>>> Hi all, >>>> >>>> M5 went out on 2026-07-21, so we're about six weeks out and main has >>>> accumulated ~30 commits since. I'd like to cut 3.0.0-M6 rather than let >>>> that pile grow further and we have a target around 08/18. >>>> >>>> What's already on main since M5: >>>> >>>> - OPENNLP-1888 Document annotation container with typed, offset-anchored >>>> layers (#1182). This is the biggest one and the one several other PRs >>>> build on, so getting it into users' hands early matters. >>>> - OPENNLP-1903 BeamSearch: chain nodes instead of per-candidate Sequence >>>> copies (#1205) >>>> - OPENNLP-1906 sentence detector abbreviation veto was quadratic in >>>> document length (#1208) >>>> - OPENNLP-1904 / -1905 default-locale case folding breaking Morfologik >>>> lookup and lemmatizer models (#1206, #1207) >>>> - OPENNLP-1922 ReDoS / stack exhaustion in the RegexNameFinderFactory >>>> EMAIL and URL patterns (#1250) >>>> - OPENNLP-1899 hardened SymSpell deserialization (#1197) >>>> - OPENNLP-1902 cached models verified against their SHA-512 checksum >>>> (#1239) >>>> - OPENNLP-1900 documented and published security model (#1198) >>>> - OPENNLP-1913 / -1915 Maven Wrapper in the source release, mvnw.cmd >>>> line endings (#1229, #1231) >>>> >>>> That's already a reasonable milestone on its own. On top of that I'd >>>> propose we try to land these four before we cut: >>>> >>>> 1. OPENNLP-1921 Paragraph normalizer for Gutenberg-style and Markdown >>>> text (#1249, tusharsaini18899). Approved already, small, and it slots >>>> into the normalizer chain we cleaned up in OPENNLP-1916. >>>> >>>> 2. OPENNLP-1893 Hunspell affix dictionaries for stemming (#1190, >>>> krickert). Classified as core in the scope thread without objection: >>>> fits the existing abstractions, no new dependencies, no bundled data. >>>> Currently "changes requested" - if that round gets addressed I'd like >>>> it in M6. (#1191, krickert). >>>> >>>> 3. OPENNLP-1909 Verified installer for user-supplied third-party >>>> resources (#1211, krickert). Interface in core, >>>> concrete downloaders in the modules they need it - M6 is a good place >>>> to >>>> try that split out in practice rather than on the list. >>>> >>>> 4. OPENNLP-1894 Dictionary-based tokenization for Japanese, Korean and >>>> Chinese (#1191, krickert); open up for Asian language feels natural in >>>> a globalized world. >>>> >>>> To avoid a misunderstanding: putting something >>>> into a milestone is not the same as deciding it belongs in core for >>>> 3.0.0 final. Milestones are explicitly not API-frozen - that is what the >>>> M is for. If CJK tokenization or the installer ships in M6 and we later >>>> agree it belongs in addons, moving it before the final release is >>>> exactly the kind of change a milestone series is supposed to absorb. I'd >>>> rather we get these in front of users and collect feedback than hold a >>>> release while the scope thread converges. >>>> >>>> So M6 does not preempt >>>> >>>> https://lists.apache.org/thread/vc8mnvfp6rmr5qb2s9y6hw0zpcxm6t68 >>>> >>>> and I'd like that thread to keep running in parallel. The open questions >>>> there - the review bar for addons, the compatibility contract, who >>>> drives discoverability - are release-cadence questions for addons, not >>>> or a core milestone. Same for the addons cleanup and getting >>>> it to Maven Central: worth doing, independent of M6. >>>> >>>> If anything on the list above isn't ready, it drops from M6 and goes >>>> into M7 (or GA???). I'd rather cut on a date than on a feature list. >>>> >>>> Objections, additions, or things I've missed that should be in M6? >>>> >>>> Gruß >>>> Richard >> >>
