tl;dr
- Slimmed down SentencePiece to addons and API in core
- proposing group id to be "org.apache.opennlp.addons" since it follows
maven convention
- changed addons documentation structure to doxia
- Opinion - we should do the same in v3.0 as it'll encourage more
documentation writers
- Opinion - I like Doxia, and it's an apache project
- Offer - I can do a first pass with identical documentation
Martin - this wouldn't be my first release management rodeo - let me know
what help you'd like first.
I created a branch in addons to serve as a canary in the coal mine. The
SentencePiece feature was slimmed down and the add-on should reside here;
this includes a few changes:
1. Marking the group ID in Maven as `org.apache.opennlp.addons`. I think
this also gives a cleaner "graduation" path where if we ever move it to
core, there's a clean delineation.
2. I changed the repo's documentation to use Doxia. In the spirit of the
addons and cleanup, our current doc manager doesn't look like it's being
maintained. So I moved the documentation to the maven-based doxia which
lets you write everything in markdown. I'm pretty certain we'd all prefer
this style for core too, because the current process feels dated. If we
want it in core, I'll volunteer to make a byte-identical manual as a PR so
we can test it out there too. So far I think it's better and stays within
the apache family of products..
3. I slimmed my "SentencePiece" PR so it's minimal for a 3.0 release
<https://github.com/apache/opennlp/pull/1165>. Less code to review and I
created a PR designed to merge into my proposed structure PR.
<https://github.com/apache/opennlp-addons/pull/178>
4. I can do more, but I'm holding off here so people can do a spot check
and confirm this is the right direction.
5. Feed free to edit the branch however anyone likes- this is just a firt
pass as I think it'll be easier to understand. Summary (that I did a few
edit passes with manually) are here.
<https://github.com/apache/opennlp-addons/blob/OPENNLP-1924-canary-addon/src/site/markdown/addon-guidelines.md>
On Tue, Sep 1, 2026 at 3:11 AM Martin Wiesner <[email protected]> wrote:
> Hi all,
>
> Thanks for all the valuable input to this discussion. I've now read most
> of it completely (which was quite a lot!) and I'like to add my 2c here
> now...
>
> (1) If we choose to release opennlp-addons, I'd be fine with that as long
> as more people step up for the role release manager so our shoulders have
> no heavy load per person. Assuming that: +1
> (2) No new deps: Yes, clearly +1 - OpenNLP 3.x was originally intended as
> modularized form of 2.x - so we should keep with as much compatibility as
> possible. New deps and too many new feature would contradict what rzo1 and
> I have proposed and envisioned back in 2025. IMHO, Lucene and other heavy
> components should be avoided at all times, as Richard has outlined with his
> argument very well.
> (3) Yes, gRPC PR could best be developed as standalone tool, therefore +1
> for own repo, given the setup + migration out of sandbox is done by the
> team. Personally, I'd focus on reviews in core / addons, however. Should be
> tackled, once OpenNLP 3.0.0 (GA) is finally out in November.
> (4) OPENNLP-1888 just landed in core/main. The rest - OPENNLP-1885, 1893,
> 1909, and maybe 1894 - should also fit and be integrated in core. I'd like
> to put effort in here so M6 is maturing soon and we can set a RC1 for it up
> maybe next week.
> (5) PRs around "embeddings" should for now live and be maintained in
> opennlp-addons. Those make a real argument towards adopting newer methods
> and components and thus could give us the signal if it is really required /
> wanted by the OpenNLP community. We can monitor download numbers for that
> easily. If agreed, I would free up the M6 release from non-core issues, so
> that the picture becomes clearer.
> (6) I like Atita's question on the review bar for addons a lot:
> Personally, I aim for high quality software, thus I'm in favor of a high
> review bar, that is documentation, tests, eval, formatting style, and so
> on. Still, I feel that innovation could be slowed down by it drastically.
> At least tests with good coverage and minimal documentation (addons
> developer manual??) plus 1 reviewing person requirement would be fine for
> me. Open for your opinions and/or concepts.
> (7) API-compat: From my perspective, opennlp-addons should strictly adhere
> to opennlp-api and its contracts. Oc, additions can extend and broaden
> features, yet should not re-invent or circumvent.
>
> Hope, it was helpful input.
> Interested in further input if others (former PMC members) want to jump in
> here.
>
> Best
> Martin | mawiesne
>
> On 2026/08/31 18:14:49 Richard Zowalla wrote:
> > Docs as part of the PRs is fine with me, go ahead with that.
> >
> > On the rest I'd still like to wait. We've had four people in this thread
> so
> > far and I don't think that's broad consensus yet. Short summary of where
> we
> > stand, in case it makes it easier for others to jump in:
> >
> > Not really contested:
> > - Clean up addons and get it releasing properly to Maven Central. Useful
> > regardless of what we decide about the split.
> > - No new dependencies in core.
> > - The gRPC PR is a PoC and gets its own build/repo, not core.
> >
> > Proposed but not yet decided:
> > - Core gets the shared foundations: Document Shape (OPENNLP-1888),
> > SentencePiece (OPENNLP-1885), and the resource loader (OPENNLP-1909),
> > with the interface in core and the downloaders in addons.
> > - Everything else goes to addons behind SPI loaders, with a narrow
> exception
> > if an addon surfaces a genuinely missing abstraction in core.
> > - Not discussed much either way: CJK tokenization (OPENNLP-1894), where
> > Kristian makes an argument for core, and the dependency parser
> (OPENNLP-547).
> >
> > Still open:
> > - The review bar for addons. Atita asked this and nobody has answered:
> does an
> > addons PR need a lighter bar than core, or the same rigor? Much of the
> rest
> > depends on it.
> > - What addons promises in terms of compatibility with core APIs, if
> anything.
> > - Who drives discoverability, since promotion from addons to core only
> works
> > if there's an adoption signal to promote on.
> > - Review capacity in general, including whether drafts should count
> against
> > the queue.
> >
> > Kristian, one correction. You wrote that for core this leaves us with
> just
> > ONNX and Lucene. It's just ONNX. We have no Lucene dependency in core,
> neither
> > on main nor on 2.x. The only Lucene in the tree is on your
> > OPENNLP-1911-vector-search-evaluation branch:
> > org.apache.lucene:lucene-core 10.4.0, test scope, in
> > opennlp-extensions/opennlp-embeddings, as the HNSW baseline for the
> search
> > evaluation. Otherwise Lucene only shows up as provenance, in the
> stopword lists and in the
> > Porter stemmer we copied in OPENNLP-337, where we dropped the Lucene
> > ArrayUtils dependency on purpose.
> >
> > To be clear about my own position: I don't want a Lucene dependency in
> core in
> > any scope. Many of our users run OpenNLP inside a Lucene, Solr or
> > Elasticsearch application, where the Lucene version is set by the host
> and not
> > by us. If core depends on Lucene, all of them get a version conflict to
> sort
> > out, in the module where we promise the most compatibility. Lucene also
> > releases on its own schedule, bumps majors and changes formats, and sets
> its
> > own Java baseline. I don't want core to follow that just to have a
> benchmark
> > baseline (so I would say we can even drop the test dependency, imho).
> >
> > Gruß
> > Richard
> >
> > > Am 31.08.2026 um 19:45 schrieb Kristian Rickert <[email protected]>:
> > >
> > > Thanks. So I think I got it - the suggested steps would go:
> > >
> > > 1) (Before Tuesday EOD + after I do the 2.x patches) I'll spruce up the
> > > documentation that covers the purpose and shape of the APIs (i.e. which
> > > features and data get attached, etc.). This way, the documentation
> will be
> > > part of the PR and summarized. I'll reply here when it's done.
> > >
> > > 2) We'll review the three shared foundations in OpenNLP core: Document
> > > Shape (OPENNLP-1888, already in progress), SentencePiece/subword
> support
> > > (OPENNLP-1885), and the file/resource loader (OPENNLP-1909).
> > >
> > > 3) Clean up addons. Assuming Martin is on board, we'll publish the
> addons
> > > as real Maven Central artifacts.
> > >
> > > All other PRs are addons via SPI loaders.
> > >
> > > Reviewer churn should transform into an availability based queue. A
> chill
> > > pace for new 3.x features if they aren't done before the 3.0 release.
> Per
> > > earlier, this is a good problem :)
> > >
> > > If an addon exposes a reusable missing abstraction (which shouldn't
> happen
> > > often), we submit only that small interface or API addition to core.
> > > Otherwise we'll need a compelling reason.
> > >
> > >
> > >
> > >
> > > On Sun, Aug 30, 2026 at 1:20 PM Atita Arora <[email protected]>
> wrote:
> > >
> > >> Thanks Richard for kicking this off, and Jeff and Kristian for the
> detail.
> > >> This is worth getting right, so I'm glad we're spending the time on
> it.
> > >>
> > >> Reading the thread, I think we're carrying three goals in one
> proposal, and
> > >> separating them helps, because they don't share a solution:
> > >>
> > >> 1. Clear the review crunch (throughput)
> > >> 2. Keep core minimal and coherent (architecture)
> > >> 3. Give experimental work a home and a path to core (adoption)
> > >>
> > >> The core/addons line mainly solves #2. It only touches #1 if addons
> carries
> > >> a genuinely lighter review bar, so I'd like us to decide that
> explicitly:
> > >> does a PR routed to addons get a lower bar (say one committer +1 or
> > >> something else), yes or no? If yes, the split buys us real review
> time. If
> > >> addons gets the same rigor as core, we get a tidier tree and the
> crunch
> > >> stays. A lot of this hinges on that answer.
> > >>
> > >> On Kristian's dependency point: if addons ships on its own cadence and
> > >> leans on core APIs, we should state the compatibility contract up
> front
> > >> rather than discover it later. Whatever we land on, let's write down
> what
> > >> addons promises, or explicitly doesn't.
> > >>
> > >> Where I'm most wary is #3. Richard's "promote when it shows adoption"
> only
> > >> works if addons is discoverable enough to produce that signal, and
> today it
> > >> isn't. So Jeff's "give addons some TLC and get it releasing" reads to
> me as
> > >> the precondition, not a nice-to-have. Without it the membrane is
> one-way:
> > >> good work goes in and nothing comes back. That's the Solr/Lucene
> contrib
> > >> pattern a lot of us have watched, plenty of good code that never got a
> > >> spotlight because nothing was driving adoption.
> > >>
> > >> So my read is it's all three, sequenced:
> > >> - Cleanup first: addons on Maven Central with real releases (Jeff's
> point).
> > >> Worth doing regardless.
> > >> - Then the line: document the core/addons rule, once we've settled the
> > >> review-bar question above.
> > >> - The gap neither of those closes: review capacity (growing the
> committer
> > >> pool, and Kristian's right that drafts shouldn't count against the
> queue)
> > >> and adoption (someone has to drive discoverability, or promotion never
> > >> fires).
> > >>
> > >> One concrete thing that would help the review load directly. For the
> > >> load-bearing pieces, the annotation layers and SentencePiece that
> most of
> > >> the rest depends on, a short design note per component family ahead
> of the
> > >> PRs would let us review the shape without reconstructing it from the
> diff.
> > >> Kristian, you've shipped a lot lately and it's substantial, and it's
> > >> frankly a lot to grasp from PR review alone. A design note also
> forces the
> > >> core-vs-addon and version questions to be answered up front, and
> doubles as
> > >> the adoption artifact (the devrel side of things! ;) ) we'd want later
> > >> anyway.
> > >>
> > >> Happy to help on the addons cleanup and release side if that's useful.
> > >>
> > >> Best,
> > >> Atita
> > >>
> > >> On Wed, Aug 26, 2026 at 5:00 AM Kristian Rickert <[email protected]>
> > >> wrote:
> > >>
> > >>> Also - my thoughts on the open pieces:
> > >>>
> > >>> 1. *(Core) Annotation layers* heavily strengthen the API and allow
> for
> > >>> add-ons. This strengthens the add-ons argument as it's 100% an
> API
> > >>> feature. Almost no PRs are useful without this landing.
> > >> (OPENNLP-1888)
> > >>> 2. *(Core) SentencePiece inference*: OPENNLP-1885 (#1165) - same as
> > >>> annotations. Without it, sentence features don't have a good home.
> > >>> Embeddings become weakened. Cascades into the other embedding
> > >> features.
> > >>> It's an interface, so a good home and like annotations, strengthens
> > >> the
> > >>> argument for add-ons.
> > >>> 3. *(Core) Dependency parser* - because why rely on python? It's
> > >> always
> > >>> bothered me that when I used OpenNLP to hybrid java and python.
> it's
> > >>> confusing, and makes me want to just use Python. This removed that
> > >>> dependency ( OPENNLP-547 (#1236) ).
> > >>> 4. *(Should be core - can be add-on - better off core) CJK
> dictionary
> > >>> tokenization*: OPENNLP-1894 (#1191) - I focused so hard on UTF as a
> > >>> precursor for this. 1/2 the world works in the CJK space, as an
> > >> Apache
> > >>> project I'd imagine we want that reach. Not only would we be more
> > >>> popular
> > >>> - it widens the language market and will attract CJK native
> > >> programmers
> > >>> to
> > >>> help us. We need that help - and this is how we can attract
> > >>> contributors. Not just a good feature, but a popularity and
> > >>> recruitment tool.
> > >>> 5. *(Separate jar) Verified resource installer*: OPENNLP-1909
> (#1211)
> > >> -
> > >>> core but maybe it's own entity. It's not even an NLP feature.
> Add-on
> > >>> or
> > >>> separate JAR. Interface in core, downloaders in add-ons.
> > >>>
> > >>>
> > >>> Add on considerations:
> > >>>
> > >>> CJK can be an add-on, but why limit support only to English/Euro
> > >>> languages? Document analysis falls apart early in the parsing
> stages -
> > >> you
> > >>> can't analyze what's going on if tokens are noisy. This can help so
> many
> > >>> fields and feels more like a first-class feature than an add-on.
> This
> > >> can
> > >>> help LLM translations and search engines alike.
> > >>>
> > >>>
> > >>> On Tue, Aug 25, 2026 at 7:22 PM Kristian Rickert <[email protected]
> >
> > >>> wrote:
> > >>>
> > >>>> Great points all around. I'd like to address them and get some
> clarity
> > >>>> regarding dependencies.
> > >>>>
> > >>>> *Dependencies*
> > >>>>
> > >>>> Currently, none of the open or draft PRs introduce new dependencies.
> > >> For
> > >>>> core, this leaves us with just ONNX and Lucene, which makes total
> > >> sense.
> > >>>> That won't change.
> > >>>>
> > >>>> I agree that addons *could* be a good place to allow new
> dependencies,
> > >>>> provided we use them sparingly. When they are necessary, we should
> use
> > >> an
> > >>>> SPI service loader over an interface so that accepting them becomes
> a
> > >>>> startup-pluggable decision. Since none of the current open PRs
> require
> > >>> new
> > >>>> dependencies, I am completely fine with a rule allowing zero new
> > >>>> dependencies for now.
> > >>>>
> > >>>> *gRPC PR*
> > >>>>
> > >>>> The gRPC PR does bring in netty, and I think early on we all agreed
> to
> > >>>> make it a separate build anyway. It is currently in a Proof of
> Concept
> > >>>> (PoC) status, it does not belong in core.
> > >>>>
> > >>>> For the server dependencies, I chose vanilla Netty because it is
> fast,
> > >>>> reference-standard, and widely adopted.
> > >>>>
> > >>>> Whenever I introduce a new dependency, I follow the SPI loading
> > >> approach
> > >>>> used by Quarkus and Micronaut: create a service interface with a
> pure
> > >> JDK
> > >>>> default implementation (e.g., saving to disk instead of S3), but
> > >> provide
> > >>>> interfaces for optional storage layers like S3 or Azure. I would
> > >>> eventually
> > >>>> like to offer the ability to save trained models from the server to
> S3,
> > >>>> Azure, or Google Cloud via a thin repository service layer, keeping
> > >>>> everything dependent on gRPC interfaces to avoid other dependencies.
> > >>>>
> > >>>>
> > >>>> *Downloads & Hosting Models*
> > >>>> LEGAL-732 <
> > >>> https://issues.apache.org/jira/projects/LEGAL/issues/LEGAL-732>
> > >>>> cleared up our current limitations, and none of the PRs are
> breaking so
> > >>>> far, so we should use that as our guideline. For new models, as
> long as
> > >>> we
> > >>>> follow those rules, we can use the downloader
> > >>>> <https://github.com/apache/opennlp/pull/1211> or host models that
> we
> > >>>> train. Since that PR is open, we can address any downloader concerns
> > >>>> directly there.
> > >>>>
> > >>>> Regarding hosting models, I have a dozen or so ideas for new models
> we
> > >>>> could host, but I would love some guidance on how we should go about
> > >>> that.
> > >>>>
> > >>>> I am a big fan of the addons approach as long as the jars are
> pushed to
> > >>>> Maven Central so they are actually usable.
> > >>>>
> > >>>> Thoughts?
> > >>>>
> > >>>> Best regards,
> > >>>> Kristian
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
