Docs as part of the PRs is fine with me, go ahead with that. On the rest I'd still like to wait. We've had four people in this thread so far and I don't think that's broad consensus yet. Short summary of where we stand, in case it makes it easier for others to jump in:
Not really contested: - Clean up addons and get it releasing properly to Maven Central. Useful regardless of what we decide about the split. - No new dependencies in core. - The gRPC PR is a PoC and gets its own build/repo, not core. Proposed but not yet decided: - Core gets the shared foundations: Document Shape (OPENNLP-1888), SentencePiece (OPENNLP-1885), and the resource loader (OPENNLP-1909), with the interface in core and the downloaders in addons. - Everything else goes to addons behind SPI loaders, with a narrow exception if an addon surfaces a genuinely missing abstraction in core. - Not discussed much either way: CJK tokenization (OPENNLP-1894), where Kristian makes an argument for core, and the dependency parser (OPENNLP-547). Still open: - The review bar for addons. Atita asked this and nobody has answered: does an addons PR need a lighter bar than core, or the same rigor? Much of the rest depends on it. - What addons promises in terms of compatibility with core APIs, if anything. - Who drives discoverability, since promotion from addons to core only works if there's an adoption signal to promote on. - Review capacity in general, including whether drafts should count against the queue. Kristian, one correction. You wrote that for core this leaves us with just ONNX and Lucene. It's just ONNX. We have no Lucene dependency in core, neither on main nor on 2.x. The only Lucene in the tree is on your OPENNLP-1911-vector-search-evaluation branch: org.apache.lucene:lucene-core 10.4.0, test scope, in opennlp-extensions/opennlp-embeddings, as the HNSW baseline for the search evaluation. Otherwise Lucene only shows up as provenance, in the stopword lists and in the Porter stemmer we copied in OPENNLP-337, where we dropped the Lucene ArrayUtils dependency on purpose. To be clear about my own position: I don't want a Lucene dependency in core in any scope. Many of our users run OpenNLP inside a Lucene, Solr or Elasticsearch application, where the Lucene version is set by the host and not by us. If core depends on Lucene, all of them get a version conflict to sort out, in the module where we promise the most compatibility. Lucene also releases on its own schedule, bumps majors and changes formats, and sets its own Java baseline. I don't want core to follow that just to have a benchmark baseline (so I would say we can even drop the test dependency, imho). Gruß Richard > Am 31.08.2026 um 19:45 schrieb Kristian Rickert <[email protected]>: > > Thanks. So I think I got it - the suggested steps would go: > > 1) (Before Tuesday EOD + after I do the 2.x patches) I'll spruce up the > documentation that covers the purpose and shape of the APIs (i.e. which > features and data get attached, etc.). This way, the documentation will be > part of the PR and summarized. I'll reply here when it's done. > > 2) We'll review the three shared foundations in OpenNLP core: Document > Shape (OPENNLP-1888, already in progress), SentencePiece/subword support > (OPENNLP-1885), and the file/resource loader (OPENNLP-1909). > > 3) Clean up addons. Assuming Martin is on board, we'll publish the addons > as real Maven Central artifacts. > > All other PRs are addons via SPI loaders. > > Reviewer churn should transform into an availability based queue. A chill > pace for new 3.x features if they aren't done before the 3.0 release. Per > earlier, this is a good problem :) > > If an addon exposes a reusable missing abstraction (which shouldn't happen > often), we submit only that small interface or API addition to core. > Otherwise we'll need a compelling reason. > > > > > On Sun, Aug 30, 2026 at 1:20 PM Atita Arora <[email protected]> wrote: > >> Thanks Richard for kicking this off, and Jeff and Kristian for the detail. >> This is worth getting right, so I'm glad we're spending the time on it. >> >> Reading the thread, I think we're carrying three goals in one proposal, and >> separating them helps, because they don't share a solution: >> >> 1. Clear the review crunch (throughput) >> 2. Keep core minimal and coherent (architecture) >> 3. Give experimental work a home and a path to core (adoption) >> >> The core/addons line mainly solves #2. It only touches #1 if addons carries >> a genuinely lighter review bar, so I'd like us to decide that explicitly: >> does a PR routed to addons get a lower bar (say one committer +1 or >> something else), yes or no? If yes, the split buys us real review time. If >> addons gets the same rigor as core, we get a tidier tree and the crunch >> stays. A lot of this hinges on that answer. >> >> On Kristian's dependency point: if addons ships on its own cadence and >> leans on core APIs, we should state the compatibility contract up front >> rather than discover it later. Whatever we land on, let's write down what >> addons promises, or explicitly doesn't. >> >> Where I'm most wary is #3. Richard's "promote when it shows adoption" only >> works if addons is discoverable enough to produce that signal, and today it >> isn't. So Jeff's "give addons some TLC and get it releasing" reads to me as >> the precondition, not a nice-to-have. Without it the membrane is one-way: >> good work goes in and nothing comes back. That's the Solr/Lucene contrib >> pattern a lot of us have watched, plenty of good code that never got a >> spotlight because nothing was driving adoption. >> >> So my read is it's all three, sequenced: >> - Cleanup first: addons on Maven Central with real releases (Jeff's point). >> Worth doing regardless. >> - Then the line: document the core/addons rule, once we've settled the >> review-bar question above. >> - The gap neither of those closes: review capacity (growing the committer >> pool, and Kristian's right that drafts shouldn't count against the queue) >> and adoption (someone has to drive discoverability, or promotion never >> fires). >> >> One concrete thing that would help the review load directly. For the >> load-bearing pieces, the annotation layers and SentencePiece that most of >> the rest depends on, a short design note per component family ahead of the >> PRs would let us review the shape without reconstructing it from the diff. >> Kristian, you've shipped a lot lately and it's substantial, and it's >> frankly a lot to grasp from PR review alone. A design note also forces the >> core-vs-addon and version questions to be answered up front, and doubles as >> the adoption artifact (the devrel side of things! ;) ) we'd want later >> anyway. >> >> Happy to help on the addons cleanup and release side if that's useful. >> >> Best, >> Atita >> >> On Wed, Aug 26, 2026 at 5:00 AM Kristian Rickert <[email protected]> >> wrote: >> >>> Also - my thoughts on the open pieces: >>> >>> 1. *(Core) Annotation layers* heavily strengthen the API and allow for >>> add-ons. This strengthens the add-ons argument as it's 100% an API >>> feature. Almost no PRs are useful without this landing. >> (OPENNLP-1888) >>> 2. *(Core) SentencePiece inference*: OPENNLP-1885 (#1165) - same as >>> annotations. Without it, sentence features don't have a good home. >>> Embeddings become weakened. Cascades into the other embedding >> features. >>> It's an interface, so a good home and like annotations, strengthens >> the >>> argument for add-ons. >>> 3. *(Core) Dependency parser* - because why rely on python? It's >> always >>> bothered me that when I used OpenNLP to hybrid java and python. it's >>> confusing, and makes me want to just use Python. This removed that >>> dependency ( OPENNLP-547 (#1236) ). >>> 4. *(Should be core - can be add-on - better off core) CJK dictionary >>> tokenization*: OPENNLP-1894 (#1191) - I focused so hard on UTF as a >>> precursor for this. 1/2 the world works in the CJK space, as an >> Apache >>> project I'd imagine we want that reach. Not only would we be more >>> popular >>> - it widens the language market and will attract CJK native >> programmers >>> to >>> help us. We need that help - and this is how we can attract >>> contributors. Not just a good feature, but a popularity and >>> recruitment tool. >>> 5. *(Separate jar) Verified resource installer*: OPENNLP-1909 (#1211) >> - >>> core but maybe it's own entity. It's not even an NLP feature. Add-on >>> or >>> separate JAR. Interface in core, downloaders in add-ons. >>> >>> >>> Add on considerations: >>> >>> CJK can be an add-on, but why limit support only to English/Euro >>> languages? Document analysis falls apart early in the parsing stages - >> you >>> can't analyze what's going on if tokens are noisy. This can help so many >>> fields and feels more like a first-class feature than an add-on. This >> can >>> help LLM translations and search engines alike. >>> >>> >>> On Tue, Aug 25, 2026 at 7:22 PM Kristian Rickert <[email protected]> >>> wrote: >>> >>>> Great points all around. I'd like to address them and get some clarity >>>> regarding dependencies. >>>> >>>> *Dependencies* >>>> >>>> Currently, none of the open or draft PRs introduce new dependencies. >> For >>>> core, this leaves us with just ONNX and Lucene, which makes total >> sense. >>>> That won't change. >>>> >>>> I agree that addons *could* be a good place to allow new dependencies, >>>> provided we use them sparingly. When they are necessary, we should use >> an >>>> SPI service loader over an interface so that accepting them becomes a >>>> startup-pluggable decision. Since none of the current open PRs require >>> new >>>> dependencies, I am completely fine with a rule allowing zero new >>>> dependencies for now. >>>> >>>> *gRPC PR* >>>> >>>> The gRPC PR does bring in netty, and I think early on we all agreed to >>>> make it a separate build anyway. It is currently in a Proof of Concept >>>> (PoC) status, it does not belong in core. >>>> >>>> For the server dependencies, I chose vanilla Netty because it is fast, >>>> reference-standard, and widely adopted. >>>> >>>> Whenever I introduce a new dependency, I follow the SPI loading >> approach >>>> used by Quarkus and Micronaut: create a service interface with a pure >> JDK >>>> default implementation (e.g., saving to disk instead of S3), but >> provide >>>> interfaces for optional storage layers like S3 or Azure. I would >>> eventually >>>> like to offer the ability to save trained models from the server to S3, >>>> Azure, or Google Cloud via a thin repository service layer, keeping >>>> everything dependent on gRPC interfaces to avoid other dependencies. >>>> >>>> >>>> *Downloads & Hosting Models* >>>> LEGAL-732 < >>> https://issues.apache.org/jira/projects/LEGAL/issues/LEGAL-732> >>>> cleared up our current limitations, and none of the PRs are breaking so >>>> far, so we should use that as our guideline. For new models, as long as >>> we >>>> follow those rules, we can use the downloader >>>> <https://github.com/apache/opennlp/pull/1211> or host models that we >>>> train. Since that PR is open, we can address any downloader concerns >>>> directly there. >>>> >>>> Regarding hosting models, I have a dozen or so ideas for new models we >>>> could host, but I would love some guidance on how we should go about >>> that. >>>> >>>> I am a big fan of the addons approach as long as the jars are pushed to >>>> Maven Central so they are actually usable. >>>> >>>> Thoughts? >>>> >>>> Best regards, >>>> Kristian >>>> >>>> >>> >>
