[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133812#comment-18133812
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6479182604
- **Branch:**
https://github.com/ai-pipestream/tika/tree/TIKA-4771-message-parser
- **Commit:** https://github.com/ai-pipestream/tika/commit/f3907bd49
- **Tim-diff:**
https://github.com/ai-pipestream/tika/compare/6480d6f33...TIKA-4771-message-parser
# tika-grpc-message-parser
Parse your own protobuf messages into Tika documents.
You already have a typed protobuf message: a reply from your own gRPC
service, a Kafka record,
a crawler's WARC record. You want it back from Tika in the same shape as any
parsed file. Write a
`MessageParser` for it, the same way you'd write a Tika parser for a file
format.
## The interface
```java
public interface MessageParser<T extends Message> {
Class<T> messageType(); // the message you read
List<Property> properties(); // your dictionary: the keys you write,
with their types
List<Metadata> parse(T message) throws TikaException;
}
```
`parse` returns what `/rmeta` returns: `[0]` is the container, and embedded
documents follow, each
with `TikaCoreProperties.EMBEDDED_ID_PATH` set. Text goes under
`TikaCoreProperties.TIKA_CONTENT`.
You do the mapping in plain Java: read the getters, set the metadata.
## An example
```java
public class WarcRecordParser implements MessageParser<WarcRecord> {
static final Property STATUS =
Property.externalInteger("warc:http-status");
public Class<WarcRecord> messageType() { return WarcRecord.class; }
public List<Property> properties() { return List.of(STATUS); }
public List<Metadata> parse(WarcRecord record) {
Metadata m = new Metadata();
m.set(TikaCoreProperties.SOURCE_PATH, record.getTargetUri());
m.set(STATUS, record.getHttpStatus());
m.set(TikaCoreProperties.TIKA_CONTENT, record.getText());
return List.of(m);
}
}
```
Put the class and your generated stubs in one jar, list the class in
`META-INF/services/org.apache.tika.grpc.spi.MessageParser`, and put the jar
on the tika-grpc
classpath. Tika never sees your `.proto`. Two parsers for one message type
stop the server at
startup.
## Calling it
`MessageParserService` in `org.apache.tika.grpc.messages.v2alpha1` runs on
the tika-grpc port. It
takes the message as a `google.protobuf.Any` and picks the parser by its
type URL:
| RPC | What you get |
|---|---|
| `ParseMessage` | every document in one `ParseMessageResponse` |
| `ParseMessageStream` | one document per message, same order |
| `ParseMessageBidi` | send many messages on one stream; one response per
`request_id`. A failed message gets `error` and the stream keeps going |
```java
ParseMessageResponse r =
MessageParserServiceGrpc.newBlockingStub(channel).parseMessage(
ParseMessageRequest.newBuilder().setMessage(Any.pack(record)).build());
```
Each `ParsedDocument` has `embedded_path`, `content`, and `metadata` as a
`google.protobuf.Struct`, so it reads as plain JSON:
```json
{ "dc:title": ["Example"], "warc:http-status": 200, "tk:chunks": [{"text":
"...", "start": 0}] }
```
Booleans and numbers are themselves, JSON keys are nested JSON, multi-valued
keys are lists, and everything else is Tika's string. A value that doesn't fit
its type stays a string. Keys in Tika's
metadata registry use the registry's type; any other key uses the type from
your `properties()`.
## Errors
| | Unary and stream | Bidi |
|---|---|---|
| no parser for the type | `NOT_FOUND` | `error.code` 5 |
| no message, bytes that don't unpack, or `parse` throws `TikaException` |
`INVALID_ARGUMENT` | `error.code` 3 |
| `parse` throws anything else | `INTERNAL` | `error.code` 13 |
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
> Attachments: 20261028-grpc-v2-design.md
>
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
