[
https://issues.apache.org/jira/browse/TIKA-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133596#comment-18133596
]
ASF GitHub Bot commented on TIKA-4795:
--------------------------------------
krickert opened a new pull request, #3449:
URL: https://github.com/apache/tika/pull/3449
Depends on #2961. Adds TikaV2Service.ParseBytes: the caller sends the bytes
it holds and gets the same Document back, with nothing fetched. It's for
crawlers that want a parse of exactly what they captured. The proto change is
additive.
Content is capped (256 MiB by default) and concurrency is capped, with
RESOURCE_EXHAUSTED before anything is spooled. When a parse fails, ParseBytes
still fills in the digest and content type, and a caller can mark content as
truncated. There's a small change to tika-core's GZipSpecializationDetector.
Design and reasoning: tika-grpc/docs/parse-bytes.md. Start with
ParseBytesRequest, admitParseBytes and ParseBytesSpool.
Tested with `mvn clean verify -pl tika-grpc-api,tika-grpc-mapper,tika-grpc`
(122 tika-grpc tests) and from StormCrawler with
https://github.com/dpol1/tika-demo.
Davide (dpol1) started this and wrote the first version.
> ParseBytes: parse-only entrypoint for callers that already hold the document
> bytes
> ----------------------------------------------------------------------------------
>
> Key: TIKA-4795
> URL: https://issues.apache.org/jira/browse/TIKA-4795
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Davide Polato
> Priority: Major
> Labels: grpc, pipes, protobuf
>
> h4. Goal
> Add a parse-only tika-grpc entrypoint that parses the exact bytes supplied by
> the caller, without asking Tika to fetch or re-fetch the resource.
> This was split from the typed {{Document}} work in
> [TIKA-4766|https://issues.apache.org/jira/browse/TIKA-4766] / [PR
> #2961|https://github.com/apache/tika/pull/2961], whose description states
> that "{{ParseBytes}} is deferred to its own JIRA."
> h4. Motivation
> {{FetchAndParse}} obtains content through a registered fetcher. Callers such
> as web crawlers have already acquired the resource bytes. Requiring another
> acquisition path can duplicate the transfer or parse a different
> representation from the one the caller observed because of redirects,
> cookies, authentication, robots and rate-limiting decisions, or transient
> content.
> The parse result should correspond to the exact representation captured by
> the caller.
> Apache StormCrawler is a concrete consumer. Its parsing bolts already receive
> the acquired bytes together with URL and protocol metadata. StormCrawler can
> contribute crawler requirements and input fixtures including truncated
> payloads, declared-vs-actual charset mismatches, malformed archives,
> documents containing embedded resources, HTML with {{<base>}}, and distinct
> URLs with identical content.
> h4. Proposed observable behavior
> This issue defines observable behavior, not the internal buffering, fetcher,
> or Pipes implementation.
> The request carries the exact byte sequence. It may also carry:
> * an optional opaque caller correlation id, echoed but never interpreted
> * source URI and effective URI when available, plus a base URI for link
> resolution; these are provenance metadata and must never be dereferenced by
> the service
> * resource name
> * declared media type and charset hints
> * optional declared content length and digest
> * a truncation flag
> Input must be bounded, and deadline or cancellation must terminate the
> associated parse work.
> The reply reuses the typed {{Document}} contract introduced by #2961 rather
> than defining a second parse-result model.
> The current direction discussed in #2961 is to place this on the experimental
> v2 surface so that it does not gate the 4.0.0 release.
> h4. Decided in the branch (2026-09-02)
> * bounded unary {{TikaV2.ParseBytes}} on the experimental v2 surface; the
> reply is the typed {{Document}} of #2961
> * provenance carried and echoed: correlation id (as {{Document.id}}, the
> caller's key), source/effective/base URI, resource name, truncation flag.
> Declared media type, charset, length and digest keep field numbers 7-10
> reserved until they have verifiable semantics
> * payload bounded by a content cap ({{grpc.parseBytesMaxContentBytes}}, 64
> MiB by default), separate from the gRPC inbound limit; bytes routed like a
> tika-server request body, inline up to {{pipes.maxInlineBytes}}, spooled
> above it
> * branch
> [TIKA-4795-parseBytes|https://github.com/ai-pipestream/tika/tree/TIKA-4795-parseBytes],
> stacked on #2961; PR once #2961 lands
> h4. Non-goals
> This issue does not define the structured content tree or its Markdown
> projection, the embedded-document representation, extension payloads such as
> {{google.protobuf.Any}}, or downstream NLP and embedding enrichment. Those
> remain separate TIKA-4766 stages and follow-up issues.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
