[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133659#comment-18133659
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6473964932
First - much better than what I did.
Second - I think we can avoid the string conversion completely
Here's a short version that builds on what you have so far:
Protobuf already has a value type, `google.protobuf.Value` (plus `ListValue`
and `Struct`). I'd use that instead of a custom `Values`. It's a first-class
type that every client + proto-nerd already knows, and you get real numbers,
booleans and lists on the wire instead of strings with hints. The struct is
smaller too. However yours is superior in one way - all numbers in Value are
doubles - so integers can lose precision.
Instead - how about this?
In java, every protobuf message extends `Message`. So the plug-in point can
be one interface, found through SPI:
```java
public interface MessageParser<T extends Message> {
Class<T> messageType();
List<Property> properties(); // the keys it writes and their types
List<Metadata> parse(T message) throws TikaException;
}
```
So you'd have to just make a plugin jar ships its own generated stubs and
one of these. The parser maps the message into Metadata the same way any Tika
parser does, in plain Java. Tika never sees the .proto, and there's no mapping
config or reflection. The service takes a `google.protobuf.Any` and picks the
parser by its type URL. It returns the same document list as FetchAndParse,
with unary, streaming and bidi calls just like yours. I have it working on top
of your PR and can share the branch.
The problem with swapping strings are the weirdo things that happen - dates
and times are annoying :) And converting to strings is really heavy on the CPU
under heavy load.
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
> Attachments: 20261028-grpc-v2-design.md
>
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
