[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133643#comment-18133643
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
tballison commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6473448128
The main missing piece is typed values. @krickert @dpol1 wdyt of this,
potentially?
```
Proposal: typed values without locking in types
v2alpha1 sends every metadata value as a string, with type and cardinality
hints from Tika's metadata registry. To spare
clients from parsing, the server can also send typed copies:
message Values {
repeated string values = 1; // always present
ValueType type = 2;
Cardinality cardinality = 3;
// set only when the hint is typed and every value parses
repeated int64 integers = 4;
repeated double reals = 5;
repeated bool booleans = 6;
repeated google.protobuf.Timestamp instants = 7; // zoned dates only
}
- Strings stay the source of truth. Typed copies are derived from them. If
a key's type changes, its typed copy moves to
another field or disappears, and the strings don't change. Type changes
are gated in tika-metadata-schema
(type-changes.json).
- Nothing is guessed. If any value fails to parse, there's no typed copy.
Dates with no zone, and date-only values, stay
strings, because a Timestamp can't represent "floating" without
inventing UTC.
- No top-level Dublin Core fields for now. With typed copies,
dcterms:created and similar keys already arrive typed in
metadata. Promoting document properties is how the format taxonomy
creeps back in.
- Structural tk: values may become fields. These describe the parse
itself, not the document, and Tika owns them.
Candidates: per-document exceptions, the content handler type, and a
"content truncated" flag. They'd stay in metadata
too, so it remains a full /rmeta mirror.
```
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
> Attachments: 20261028-grpc-v2-design.md
>
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
