[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133866#comment-18133866
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6483269403
So let me know if this approach is clear. It's going to be a similar
pattern I use for the grpc output we do for tika in OpenNLP - we'll have a
document shape in there that'll get all the RAG folks excited including putting
embeddings on the data (I'll build a plugin for you as well that decorates NLP
data all over the place in Tika once we release 3.0).
That being said, I imagine we would make OpenNLP's enhancements a direct
integration, or we can make it so it's either-or (if they want to scale the
OpenNLP work, they can use the gRPC and if they want to just directly integrate
it they can do that as well).
But the idea really falls around SPI and the ability to inject services into
java now without reflection. Also, the fact that all protobufs carry the
"Message" inheritance.
FastWARC's grpc server is a good example - I can see a lot of Tika people
having WARC files. But my intent with that parser is just to have a useful
example. I can pick another service if you want as a demonstration.
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
> Attachments: 20261028-grpc-v2-design.md
>
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
