Hi all,
we currently have ~20 open PRs, most of them contributing substantial new
functionality: stemmers, subword tokenization, embeddings and vector
indexes, a gazetteer/geocoder, a lexical knowledge base, a dependency
parser, and more. First of all: this is a great problem to have.
The breadth of this work would move OpenNLP much closer to what people
expect from the Python NLP ecosystem, and I'd like us to find a home
for as much of it as possible.
That said, I don't think all of it belongs in core, and we've never
written down where the line is. Each PR currently triggers the same
case-by-case debate, which is slow for us and frustrating for
contributors. I'd like us to agree on a simple test and make it a
documented rule.
My proposal:
A contribution belongs in *core* if it:
- fits our existing abstractions / component model,
- introduces no new heavy runtime dependencies, and
- bundles no data files (models, dictionaries, datasets).
A contribution belongs in *addons* (or sandbox) if it:
- brings its own dependencies,
- bundles or downloads data, or
- introduces a whole new component family that stands on its own.
Applying that test to the current queue, as a starting point for
discussion:
Core (fits existing abstractions, no new deps/data):
- OPENNLP-1903 BeamSearch optimisation (#1205)
- OPENNLP-1886 UniNE stemmer tiers (#1166)
- OPENNLP-1893 Hunspell affix stemming (#1190)
- smaller fixes like OPENNLP-1916 (#1235)
Addons (own data and/or new component families):
- OPENNLP-1879 Gazetteer/geocoder with bundled Natural Earth data
(#1154)
- the embedding + vector index stack: OPENNLP-1877, -1895, -1910,
-1911, -1897 (#1152, #1213, #1214, #1215, #1212)
- the lexical knowledge base / WordNet work: OPENNLP-1880, -1887
(#1155, #1167)
Needs discussion (borderline under the test above):
- Document annotation layers: OPENNLP-1888, -1919, -1920 (#1182,
#1237, #1238) (my POV is: would love to have it)
- Dependency parser: OPENNLP-547 (#1236)
- SentencePiece inference: OPENNLP-1885 (#1165)
- CJK dictionary tokenization: OPENNLP-1894 (#1191)
- Verified resource installer: OPENNLP-1909 (#1211)
Two important points that go with this:
1. "Addons" must not mean "graveyard". Redirecting work there is only
good if we actually release addons on a predictable cadence. So
part of this proposal is that we commit to regular addons releases
(and probably clean up the addons repo a bit first). I'd rather we
agree on that commitment here than quietly park stuff.
2. The line must be permeable IMHO. If an addons module sees
real adoption (downloads, traffic, user questions), that's exactly
the signal to promote it into core later. Starting in addons is not
a rejection, it's an incubation path with a much lower review bar
and a faster way to get code into users' hands.
To be clear: this is not about declining contributions. It's about
matching our review capacity (which is honestly the bottleneck right
now - a handful of us cannot review 200k added lines in core in a
reasonable time frame) to a structure where good work lands somewhere
visible and released, instead of sitting in the queue for months.
If we get rough consensus on the test itself, I'd follow up with:
- a short section in CONTRIBUTING.md documenting the rule,
- triaging the open PRs accordingly together with the authors,
- a plan (separate thread) for the addons release cadence.
Looking forward to your thoughts - including, explicitly, from the
authors of the PRs above.
Gruß
Richard
