Kristian Rickert created OPENNLP-1888:
-----------------------------------------
Summary: FEEDBACK / DESIGN : creating a Document annotation
container: typed, offset-anchored layer s so every future analysis result is an
additive 3.x feature
Key: OPENNLP-1888
URL: https://issues.apache.org/jira/browse/OPENNLP-1888
Project: OpenNLP
Issue Type: Improvement
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Designing the document generic type in Java:
To be explicit, because it is the first question this proposal should answer:
*this ticket deprecates nothing and replaces nothing.* It is something I've
brought up in the chats - the importance of now being the time to land API
changes.
h2. Motivation
OpenNLP's Java API is tool-by-tool: a tokenizer returns spans, a tagger returns
strings, a name finder returns spans, and the caller reassembles the pieces by
hand, differently in every application. Each new capability adds another entry
point and result type, so the cost compounds. I ran into this head-on writing
the gRPC layer, which is mostly mapping tool outputs into a common document
structure that the Java API itself never defines. Since 3.0 has not landed yet,
this feels like the right time to discuss adding that layer.
We already have two data points. {{Term}} validated the layered-container idea,
but it is token-scoped, string-valued, and keyed by a closed enum, so it cannot
hold anything structural. The emoji annotation layer (OPENNLP-1870) is the
first structured result that did not fit, and it correctly became a separate
accessor beside {{{}Term{}}}. Every future structured result (dependency arcs,
coreference, relations, money, PII) would repeat that pattern, each growing its
own parallel surface, while the combined-document view still would not exist.
The fix is one small container in {{{}opennlp-api{}}}: typed annotation layers
over the original text, identified by open-ended keys rather than an enum. Any
future capability then becomes three additions that touch no existing type: a
payload record, a layer key, and an annotator. Since anything published in 3.x
is frozen until 4.0, shipping this in 3.0 is what keeps the parallel surfaces
from ever accumulating.
h2. Design
Four public types, all in a new package in {{{}opennlp-api{}}}:
{code:java}
public interface Document {
CharSequence text(); // the original text, never a
normalized form
<T> List<Annotation<T>> get(LayerKey<T> layer); // empty list when the layer
is absent
Set<LayerKey<?>> layers();
}
public record Annotation<T>(Span span, T value) { }
public final class LayerKey<T> {
public static <T> LayerKey<T> of(String id, Class<T> type) { ... }
}
public interface DocumentAnnotator {
Document annotate(Document document); // returns a new document
with layers added
Set<LayerKey<?>> requires();
Set<LayerKey<?>> provides();
}
{code}
Standard keys for today's results (sentences, tokens, tags, chunks, entities)
ship with thin annotators wrapping the existing task interfaces, which stay
exactly as they are. A {{DocumentAnalyzer}} builder assembles annotators into a
pipeline ordered by {{{}requires(){}}}/{{{}provides(){}}}, mirroring
{{{}TermAnalyzer{}}}.
Rules that protect the additive property:
* Spans always point into the original text; normalization maps back through
{{{}Alignment{}}}.
* Layer identity is an open key, never an enum; payload types live in the
producing feature's package, and the container never imports or enumerates them.
* Annotations reference other annotations by layer and index (a dependency arc
names token indices), keeping documents immutable and mappable to any encoding.
* No new dependencies, and no serialization framework in {{{}opennlp-api{}}};
encodings live where their dependencies already are.
* Evolution by default methods and new types only; nothing sealed, no abstract
classes at the seam.
h2. Coexistence with the task-level API
The task interfaces are not a legacy layer. They remain the primary API for
single-task use, with no deprecation and no planned removal; annotators
delegate to them, so there is one behavior reachable two ways (the same
relationship {{TermAnalyzer}} has to the normalizer rungs). Deprecation is
reserved for the rare surface a layer supersedes outright, and even then
removal waits for a major, so existing call sites compile and run for the life
of 3.x. Models are unaffected: nothing about training or decoding changes.
h2. Validation before freeze
# The existing analyze pipeline (sentences, tokens, tags, entities) expressed
as annotators, proving the adapters and ordering model.
# Dependency parsing as the first graph-shaped layer: its values reference
other annotations, so it exercises the cross-layer rules that coreference and
relation extraction will inherit. A transition-based parser over the existing
maxent machinery, trained from CoNLL-U treebanks, suffices and is a wanted
feature in its own right.
h2. Out of scope
Downstream features themselves (each is its own ticket), changes to any
existing task interface or result type, and document serialization.
h2. Acceptance criteria
* Container types in {{{}opennlp-api{}}}, no new dependencies, full javadoc;
existing tools populate a document through adapters unchanged.
* Both validation consumers pass tests covering cross-layer references and
span fidelity, including a normalization-shifted case.
* A new layer can be added in a test without touching the container's package.
* Binary compatibility from the previous release is verified mechanically in
CI.
* No existing type is modified or deprecated; the task-level API remains fully
supported.
h4. Guardrail Note:
The container's package must never grow knowledge of specific layers. The
moment it imports a payload type or enumerates layer identities, the additive
property is lost and every future feature becomes an API change again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
