Sounds good to me.
On 2026/09/02 19:21:27 Kristian Rickert wrote: > It does - but not only embeddings - > > OpenNLP needs SentencePiece to run SentencePiece-backed neural annotators > over sentences - like our own onnx features - while preserving exact > offsets into the original text. This is the main reason I wrote it. All > analysis, even in Python packages in this area does not handle this well > (if at all), making it hard to create truly accurate highlighters without > it. However, that analysis and offset tracking belongs in a core API > layer. So it's not adding a dependency; it's just an interface that tracks > the offsets for it and perhaps some associated metadata at that level. > > I'll take a second look at SentencePiece and carve out the API/core stuff; > the rest goes into addon. > > That's the reasoning: embeddings and SentencePiece are already central to > much NLP work, so the interface can exist there. As new technologies emerge > to fill this gap, we can pivot (like how the gRPC server uses OpenVINO, > TEI, or Model2Vec, for example). > > Thoughts? > > > On Tue, Sep 1, 2026, 9:02 AM Richard Zowalla <[email protected]> wrote: > > > Hss todo with embeddings, no? > > That would contract in Martin mail because he wrote "PRs around > > "embeddings" should for now live and be maintained in opennlp-addons.“. > > > > Gruß > > Richard > > > > > Am 01.09.2026 um 14:33 schrieb Kristian Rickert <[email protected]>: > > > > > > Quick clarification: > > > > > > In the core/addons thread, Martin wrote that OPENNLP-1885, OPENNLP-1893, > > > OPENNLP-1909, and possibly OPENNLP-1894 should fit in core, and that he > > > wanted to put effort into them so M6 could mature. > > > > > > The M6 proposal includes OPENNLP-1893, OPENNLP-1909, and OPENNLP-1894, > > but > > > not OPENNLP-1885. Since the two messages were posted about a minute > > apart, > > > I suspect they may have crossed? > > > > > > > > > > > > > > > > > > On Tue, Sep 1, 2026 at 3:12 AM Richard Zowalla <[email protected]> wrote: > > > > > >> Subject: [DISCUSS] 3.0.0-M6 release > > >> > > >> Hi all, > > >> > > >> M5 went out on 2026-07-21, so we're about six weeks out and main has > > >> accumulated ~30 commits since. I'd like to cut 3.0.0-M6 rather than let > > >> that pile grow further and we have a target around 08/18. > > >> > > >> What's already on main since M5: > > >> > > >> - OPENNLP-1888 Document annotation container with typed, offset-anchored > > >> layers (#1182). This is the biggest one and the one several other PRs > > >> build on, so getting it into users' hands early matters. > > >> - OPENNLP-1903 BeamSearch: chain nodes instead of per-candidate Sequence > > >> copies (#1205) > > >> - OPENNLP-1906 sentence detector abbreviation veto was quadratic in > > >> document length (#1208) > > >> - OPENNLP-1904 / -1905 default-locale case folding breaking Morfologik > > >> lookup and lemmatizer models (#1206, #1207) > > >> - OPENNLP-1922 ReDoS / stack exhaustion in the RegexNameFinderFactory > > >> EMAIL and URL patterns (#1250) > > >> - OPENNLP-1899 hardened SymSpell deserialization (#1197) > > >> - OPENNLP-1902 cached models verified against their SHA-512 checksum > > >> (#1239) > > >> - OPENNLP-1900 documented and published security model (#1198) > > >> - OPENNLP-1913 / -1915 Maven Wrapper in the source release, mvnw.cmd > > >> line endings (#1229, #1231) > > >> > > >> That's already a reasonable milestone on its own. On top of that I'd > > >> propose we try to land these four before we cut: > > >> > > >> 1. OPENNLP-1921 Paragraph normalizer for Gutenberg-style and Markdown > > >> text (#1249, tusharsaini18899). Approved already, small, and it slots > > >> into the normalizer chain we cleaned up in OPENNLP-1916. > > >> > > >> 2. OPENNLP-1893 Hunspell affix dictionaries for stemming (#1190, > > >> krickert). Classified as core in the scope thread without objection: > > >> fits the existing abstractions, no new dependencies, no bundled data. > > >> Currently "changes requested" - if that round gets addressed I'd like > > >> it in M6. (#1191, krickert). > > >> > > >> 3. OPENNLP-1909 Verified installer for user-supplied third-party > > >> resources (#1211, krickert). Interface in core, > > >> concrete downloaders in the modules they need it - M6 is a good place > > >> to > > >> try that split out in practice rather than on the list. > > >> > > >> 4. OPENNLP-1894 Dictionary-based tokenization for Japanese, Korean and > > >> Chinese (#1191, krickert); open up for Asian language feels natural in > > >> a globalized world. > > >> > > >> To avoid a misunderstanding: putting something > > >> into a milestone is not the same as deciding it belongs in core for > > >> 3.0.0 final. Milestones are explicitly not API-frozen - that is what the > > >> M is for. If CJK tokenization or the installer ships in M6 and we later > > >> agree it belongs in addons, moving it before the final release is > > >> exactly the kind of change a milestone series is supposed to absorb. I'd > > >> rather we get these in front of users and collect feedback than hold a > > >> release while the scope thread converges. > > >> > > >> So M6 does not preempt > > >> > > >> https://lists.apache.org/thread/vc8mnvfp6rmr5qb2s9y6hw0zpcxm6t68 > > >> > > >> and I'd like that thread to keep running in parallel. The open questions > > >> there - the review bar for addons, the compatibility contract, who > > >> drives discoverability - are release-cadence questions for addons, not > > >> or a core milestone. Same for the addons cleanup and getting > > >> it to Maven Central: worth doing, independent of M6. > > >> > > >> If anything on the list above isn't ready, it drops from M6 and goes > > >> into M7 (or GA???). I'd rather cut on a date than on a feature list. > > >> > > >> Objections, additions, or things I've missed that should be in M6? > > >> > > >> Gruß > > >> Richard > > > > >
