Got no capacity (for this) atm. Best Martin
> Am 21.07.2026 um 11:18 schrieb Richard Zowalla <[email protected]>: > > Any additional input? > > On 2026/07/17 13:55:56 Kristian Rickert wrote: >> Everyone: >> >> The draft PR (#1182) now reflects all 4 points, one commit per point so >> each is reviewable on its own. >> >> tl;dr - Everything promised in the last mail is now spec text plus >> enforcing code plus contract tests, nothing is prose-only. >> >> 1. Namespaced ids (86f4bee). Every key the toolkit defines carries the >> "opennlp:" prefix: opennlp:sentences, opennlp:tokens, opennlp:pos, >> opennlp:entities, and opennlp:lemmas / opennlp:stems on the adapters. >> Extensions use their own prefix, bare ids stay legal for application-local >> layers. The rule is stated on Layers, on LayerKey, and in the manual >> chapter. >> 2. Per-key scope (6c38c23). LayerKey now declares POSITIONAL or DOCUMENT. >> Positional is the default and guarantees a span on every annotation; >> LayerKey.document(id, type) creates a key for whole-document values >> (language id, category distributions, provenance), and Annotation.of(value) >> builds the span-less entry. The container polices it in both directions at >> insertion, naming the layer: a span-less annotation under a positional key >> is rejected, a spanned one under a document key likewise. One consequence >> you should know before commenting: key equality is now (id, type, scope), >> so a positional and a document key with the same id and type are distinct >> layers, same as the existing id-with-different-type rule. >> 3. Invariants (a657c00). Transcribed onto the Document interface and into >> the manual: insertion order is preserved and never reordered, layers are >> immutable once added and detached from the caller's input list, add is >> once-only operation that rejects duplicates by naming the key, therefore >> index references stay valid for the document's lifetime. The wording >> matches what the contract tests enforce, nothing new was invented for the >> prose. >> 4. Gold versus predicted (48321ff). Documented as the convention on Layers >> and in the manual: gold:opennlp:tokens beside opennlp:tokens, and a >> contract test pins that both live independently and neither can replace the >> other. >> >> Thoughts are encouraged for the draft "opennlp:" prefix itself (versus >> something dotted) and the gold:opennlp:tokens double-prefix shape. I >> prefer the ":" as the "." convention feels less of a separation. >> >> This is a great moment to discuss, as it'll establish a new convention. >> >> - Kristian >> >> On Thu, Jul 16, 2026 at 1:58 PM Kristian Rickert <[email protected]> wrote: >> >>> Hi Richard, >>> >>> Thanks for doing the survey before forming an opinion; the convergence you >>> found is the best argument the design could get. I am adopting all four >>> points. Short version first: >>> >>> tl;dr - Spans are baked in by construction. The only span-less things are >>> whole-document facts, and those are declared at the key level, not skipped >>> at return time. >>> >>> 1. Key identity. Equality is the (id, type) pair, pinned by contract >>> tests: same id and type address one layer, same id with a different type is >>> a distinct coexisting layer. Standard keys get namespaced ids before the >>> freeze ("opennlp:" prefix, extensions use their own prefix, bare ids stay >>> legal for application-local layers). >>> >>> 2. Annotations without spans. The span is structurally mandatory today: >>> the annotation record rejects a null span and the container rejects >>> out-of-bounds spans, so no code path returns a span-less result. The census >>> of every annotator built so far: >>> >>> >>> +---------------------------------+--------------------------------------+ >>> | Annotator | Span anchors on >>> | >>> >>> +---------------------------------+--------------------------------------+ >>> | Sentence / tokenizer adapters | detected spans; tokens shifted to >>> | >>> | | document coordinates >>> | >>> | POS / lemmatizer / stemmer | each token's span >>> | >>> | Name finder | the entity span >>> | >>> | Glossary | the match span >>> | >>> | PII | the mention span >>> | >>> | Coreference (chains) | each member mention's span >>> | >>> | Money / quantity / temporal / | the mention span >>> | >>> | currency conversion | >>> | >>> | Document date | the electing mention's span >>> | >>> | Geocoding (locations) | the resolved mention's span >>> | >>> | Containment (hierarchy) | the location mention's span >>> | >>> | Dependencies | the dependent token's span >>> | >>> | Relations | the covering span of both arguments >>> | >>> | Embeddings | the source annotation's span >>> | >>> >>> +---------------------------------+--------------------------------------+ >>> >>> Only one layer spans the document as a whole (the region ballot, currently >>> on a whole-document span). Proposal: scope is declared per key, never per >>> annotation. Positional keys guarantee a span on every entry; >>> document-scoped keys carry values without spans (language id, categories, >>> provenance, the ballot). That is the UIMA offset-bearing versus offset-free >>> split you cited, at library weight, and document-scoped layers staying >>> lists fits the real payloads (a categorizer returns a scored distribution). >>> >>> 3. Invariants. All three are implemented and contract-tested, and the spec >>> will state them: insertion order is preserved and never sorted; layers are >>> immutable once added and detached from the caller's input; providing an >>> existing layer is rejected loudly, add is once-only. Index references >>> (dependencies point at token positions) therefore stay valid for the >>> document's lifetime. >>> >>> 4. Gold versus predicted. Documented as a convention on the same namespace >>> scheme (gold:opennlp:tokens beside opennlp:tokens); with the once-only >>> rule, competing versions always live under distinct keys and never replace >>> each other. >>> >>> Acceptance criterion adopted, and it is already an observation rather than >>> a promise: the feature branches on the PoC fork add all of the layers above >>> without a single edit to the container package. >>> >>> Given this feedback, mind if I fold points 1 through 4 into the spec text? >>> Not finalizing, just updating. >>> >>> Rungs, >>> Kristian >>> >>> On Thu, Jul 16, 2026 at 1:22 PM Richard Zowalla <[email protected]> wrote: >>> >>>> Thanks for the summary. I did some homework on how other systems solve >>>> this before forming an opinion. Short version: the proposed design is not >>>> novel architecture. That is its strength. The same pattern has emerged >>>> independently in at least four mature systems. >>>> >>>> INCEpTION defines no container of its own. It sits on the UIMA CAS >>>> [1][2]. The CAS holds the original text and typed feature structures with >>>> begin and end offsets into it. That is classic standoff annotation, the >>>> same rule the proposal states for spans. What INCEpTION calls a layer is a >>>> UIMA type plus configurable behaviors. It distinguishes span layers, >>>> relation layers, chain layers for coreference, and metadata layers without >>>> offsets. The difference is weight. UIMA declares its type system in XML >>>> descriptors and carries reflection and serialization machinery with it. The >>>> proposal is deliberately the lightweight end of that spectrum. For a >>>> library, as opposed to an annotation platform, that is the right decision, >>>> IMHO. >>>> >>>> Stanford CoreNLP is the closest match (to us), almost line for line. Its >>>> Annotation is a type safe heterogeneous map in the sense of Bloch >>>> [3][4][5]. Keys are class objects parameterized by the value type. The >>>> container never enumerates keys. Any module adds a new key without touching >>>> the core. The proposed LayerKey is the same pattern with string identity >>>> instead of class identity. That design has held up in Standford's CoreNLP >>>> for over fifteen years. >>>> >>>> spaCy has one owning Doc object, spans as views into it, and doc.spans as >>>> a dict of named span groups under open string keys [6]. Pipeline components >>>> declare assigns and requires metadata, and the pipeline analyzer validates >>>> ordering. That is a direct precedent for the proposed requires and provides >>>> methods. GATE uses named annotation sets with untyped feature maps [7]. It >>>> is open ended like the proposal but stringly typed in its values, which the >>>> Class parameter in LayerKey correctly avoids. >>>> >>>> So the field has converged four times on the same shape: standoff spans >>>> over immutable original text, open keyed typed layers, a container ignorant >>>> of payloads, and pipelines ordered by declared dependencies. >>>> The guardrail note in the ticket, that the doc container must never learn >>>> about specific layers, is the single most important sentence in it. >>>> >>>> Four points where the precedents diverge and where I would tighten the >>>> spec before the freeze: >>>> >>>> 1. Key identity. CoreNLP gets collision free identity from class objects. >>>> String ids do not. Two extensions can both mint "entities". The spec should >>>> define key equality and adopt namespaced ids for the standard keys, IMHO. >>>> 2. Annotations without spans. Language id, document categories, >>>> provenance. Every precedent grew a home for these. UIMA has types without >>>> offsets, spaCy has doc level attributes, INCEpTION has metadata layers. >>>> Retrofitting this later creates exactly the parallel surface the proposal >>>> wants to prevent, so we should think about that upfront. >>>> 3. Index based references need stated invariants. UIMA and CoreNLP >>>> reference annotations by object identity. Indices are better for >>>> immutability and serialization, but only if layer order is defined and >>>> existing layers are never mutated or reordered. The spec should - >>>> therefore- also say what happens when an annotator provides a layer that >>>> already exists. This needs to be defined. >>>> 4. Gold versus predicted layers: GATE annotation sets and UIMA views >>>> exist because corpora might carry competing versions of the same layer. >>>> Open string keys already handle this. The convention should simply be >>>> documented. >>>> >>>> At a first glance, the validatio plan from the Jira looks right. >>>> Dependency parsing is precisely the consumer that stresses cross layer >>>> references, and coreference will inherit the same rules. >>>> I would also make the acceptance criterion, that a new layer can be added >>>> without touching the container package. >>>> >>>> Gruß >>>> Richard >>>> >>>> [1] INCEpTION Developer Guide. >>>> https://inception-project.github.io/releases/26.8/docs/developer-guide.html >>>> [2] Klie JC, Bugert M, Boullosa B, Eckart de Castilho R, Gurevych I. The >>>> INCEpTION Platform: Machine Assisted and Knowledge Oriented Interactive >>>> Annotation. COLING 2018. https://aclanthology.org/C18-2002.pdf >>>> [3] TypesafeMap. Stanford CoreNLP API. >>>> https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/util/TypesafeMap.html >>>> [4] Annotation. Stanford CoreNLP API. >>>> https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/pipeline/Annotation.html >>>> [5] CoreAnnotation. Stanford CoreNLP API. >>>> https://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/ling/CoreAnnotation.html >>>> [6] spaCy Doc API. https://spacy.io/api/doc >>>> [7] GATE Documentation, Language Resources: Corpora, Documents and >>>> Annotations. https://gate.ac.uk/sale/tao/splitch5.html >>>> >>>>> Am 15.07.2026 um 21:33 schrieb Kristian Rickert <[email protected]>: >>>>> >>>>> I accidentally sent a draft. To clear things up, I’ve opened >>>> OPENNLP-1888 >>>>> to spark a design discussion around introducing a generic Document type >>>>> into the 3.0 API. >>>>> >>>>> Currently, our Java API is tool-by-tool; as we look toward adding more >>>>> complex, structured results (like dependency arcs or relations), >>>> continuing >>>>> this pattern means we'll end up building parallel surface areas for >>>> every >>>>> new capability. >>>>> >>>>> The proposal in the JIRA ticket aims to solve this by introducing a >>>> small, >>>>> unified container in opennlp-api that uses typed annotation layers over >>>> the >>>>> original text. I want to emphasize right away that this deprecates and >>>>> replaces absolutely nothing in the existing task-level API; it simply >>>>> provides a clean, additive way to handle combined-document views moving >>>>> forward. I'd really love it if we could take a look at the full >>>>> architecture, constraints, and motivations in the ticket and share your >>>>> thoughts. >>>>> >>>>> I think this design covers what you'd see in other APIs as well as some >>>> new >>>>> frontier ideas. >>>>> >>>>> Thoughts? >>>>> >>>>> On Wed, Jul 15, 2026 at 3:17 PM Richard Zowalla <[email protected]> >>>> wrote: >>>>> >>>>>> Did this mail get mangled? >>>>>> >>>>>>> Am 15.07.2026 um 00:17 schrieb Kristian Rickert <[email protected] >>>>> : >>>>>>> >>>>>>> Two main points: >>>>>>> >>>>>>> - OPENNLP-1888 has not been started; it was created as a design >>>>>>> discussion regarding a generic document shape I'm recommending. >>>> Details >>>>>> in >>>>>>> the ticket >>>>>>> - Thank you everyone for the detailed feedback. I'll ensure every PR >>>> has >>>>>>> a section in the documentation. >>>>>>> >>>>>>> OPENNLP-1888 was opened to discuss a generic Document type for the new >>>>>> API >>>>>>> interface. Details are in the ticket, but here is the initial shape >>>> and I >>>>>>> would like anyone's thoughts on it: >>>>>>> >>>>>>> public interface Document { >>>>>>> CharSequence text(); // the original text, >>>>>>> never a normalized form >>>>>>> <T> List<Annotation<T>> get(LayerKey<T> layer); // empty list when the >>>>>>> layer is absent >>>>>>> Set<LayerKey<?>> layers(); >>>>>>> } >>>>>>> >>>>>>> public record Annotation<T>(Span span, T value) { } >>>>>>> >>>>>>> public final class LayerKey<T> { >>>>>>> public static <T> LayerKey<T> of(String id, Class<T> type) { ... } >>>>>>> } >>>>>>> >>>>>>> public interface DocumentAnnotator { >>>>>>> Document annotate(Document document); // returns a new >>>>>> document >>>>>>> with layers added >>>>>>> Set<LayerKey<?>> requires(); >>>>>>> Set<LayerKey<?>> provides(); >>>>>>> } >>>>>> >>>> >>>> >>
