[
https://issues.apache.org/jira/browse/TIKA-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert closed TIKA-4772.
----------------------------------
Resolution: Fixed
> Stream parsed-document events over gRPC; unary Document becomes the fold of
> the stream
> --------------------------------------------------------------------------------------
>
> Key: TIKA-4772
> URL: https://issues.apache.org/jira/browse/TIKA-4772
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Reporter: Kristian Rickert
> Priority: Major
>
> h3. Motivation
> Tika's parsers are already incremental: every parser emits its output
> progressively through the {{ContentHandler}} contract as it reads the input.
> The buffered single-reply gRPC response is an artifact of the transport, not
> of Tika's model. Exposing the parse as an event stream (a) matches what the
> parsers already do, (b) gives time-to-first-block latency for
> RAG/embedding/NLP pipelines that can begin work while a large document is
> still parsing, (c) bounds client memory on huge documents, and (d) makes
> partial results on failure precise instead of best-effort.
> h3. Design: one contract, two delivery modes
> {code:java}
> rpc FetchAndParseEvents(FetchAndParseRequest) returns (stream DocumentEvent)
> {}
> message DocumentEvent {
> oneof event {
> DocumentEnvelope envelope = 1; // first event: id, content_type, origin
> BlockEvent block = 2; // one content Block, in document order
> DocumentMetadata metadata = 3; // emitted when known (often late in the
> parse)
> MetadataField extra = 4;
> ExtensionResult extension = 5; // external parser results, forwarded as
> they arrive
> ParseStatus status = 6; // terminal event
> }
> }
> message BlockEvent {
> repeated int32 embedded_path = 1; // [] = root document; [2] = third
> embedded child
> Block block = 2;
> }
> {code}
> Events are append-only immutable facts (a late {{metadata}} event completes,
> never revises). The defining invariant: *The fold of the event stream is
> exactly the {{Document}} from TIKA-4766.* Backward compatibility falls out by
> construction rather than by discipline: the existing unary-style reply is
> implemented AS the fold – the server consumes its own event stream to
> completion and emits the assembled {{Document}}. There is one parse path, so
> the streamed and materialized forms cannot drift, and existing clients see
> identical behavior forever. A mid-parse failure terminates the stream with
> {{PARTIAL}}, and the fold yields a Document holding exactly the blocks
> parsed before the failure -- {{PARTIAL}} becomes a precise statement instead
> of a best-effort flag.
> h3. The event model is the wire's native model
> {{{}DocumentEvent{}}}/{{{}Block{}}} is the contract's native representation.
> Tika's existing parsers keep their ContentHandler contract exactly as-is: a
> bridge insideoutput into document events at the edge, and a reverse adapter
> serves anything that consumes ContentHandlers today – nothing about the Java
> parser APIchanges, now or later. The same event stream can equally be
> produced directly, by server components or (via the external parser
> mechanism, TIKA-4771) by services in any language, so every producer is
> first-class and identical on the wire. Structure the block model cannot
> express maps to {{{}HtmlBlock{}}}/metadata, never silently dropped.
> This extends a pattern tika-grpc already has: the server currently brokers
> registered fetchers, emitters, and pipes iterators; this applies the same
> brokering to parse output.
> h3. Phasing
> # *Phase 1 (contract + cheap wins):* declare the RPC; serve a coarse but
> valid stream (envelope, then blocks, then metadata, then terminal status);
> implement the unary-style reply as the fold. Streaming results from external
> parsers ({{{}ParseStream{}}} in TIKA-4771) also land here - the gRPC server
> calls those services directly, so forwarding their streams touches nothing
> outside tika-grpc.
> # *Phase 2 (fine granularity):* frame {{DocumentEvent}} across the
> tika-pipes forked-worker boundary so events flow live during the parse. The
> worker protocol speaks {{DocumentEvent}} frames; the existing parser fleet
> feeds it through the {{ContentHandler}} bridge at the edge. This touches
> tika-pipes-core and deserves its own discussion - maintainer input explicitly
> wanted on whether/how the worker framing should support incremental events.
> Because unary == fold, improving granularity later changes neither the
> contract nor unary behavior.
> h3. Open questions
> # Pipes worker protocol: appetite and preferred shape for incremental event
> framing across the fork boundary (this is the phase-2 gate)?
> # Ordering/assembly: block order per {{embedded_path}} is document order; is
> that sufficient, or do we want explicit sequence numbers for resumability?
> # Flow control: rely on gRPC per-stream backpressure, or add a server-side
> high-water mark with a documented buffering policy?
> # Metadata timing: emit known-early fields (content type, origin) in the
> envelope and the rest at completion, or allow multiple cumulative
> {{metadata}} events?
> h3. Relationship to other work
> Depends on TIKA-4766 (the {{{}Document{}}}/{{{}Block{}}} contract this
> streams). Relates to TIKA-4771 (external parsers; their streamed results ride
> this stream as {{extension}} events). Phase 2 will spawn a linked tika-pipes
> issue once there is agreement on direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
