[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134106#comment-18134106
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6493009529
To be more direct: it's not one per Tika parser. A {{MessageParser}} is
keyed by the incoming protobuf message type, and it's only for users who
already have a typed message (a WARC record, a Kafka record, a reply from their
own service) and want it back as Tika documents. File parsing through
FetchAndParse is unchanged.
The jar ships its own generated stubs and decides how its message maps onto
Tika's document shape. Tika never has to know the message's schema, and nothing
goes through strings.
h4. Sidenote: CEL (not planned in this ticket - not slop!)
For automated mapping later, CEL (Common Expression Language) is the
protobuf-native option. Expressions are type-checked against the message
descriptors and run directly on typed messages, so there's no protobuf to JSON
to protobuf round trip (my LLMs tried to offer JSON hacks often - steer them
away from that it can get quadratic-level slow). The closest comparison is CSS
selectors, but for protobuf fields and faster. There's a Java implementation
(cel-java).
**That's probably too big for this iteration**. A hand-written mapper keeps
the strong typing and is a lot faster than JSON, and we can add mapping and
ETL-style behavior to the v2 shape later (note: with this shape, we can add it
for later w/o needing a new version).
Mappings and transforms aren't really part of tika (at this point) - the
other parsers tend to do their mappings in a similar way. I'm suggesting for
later is an API addition in the future where you can offer this and it'll be
fast.
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
> Attachments: 20261028-grpc-v2-design.md
>
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
