[
https://issues.apache.org/jira/browse/CAMEL-27329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-27329.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/30550 (merged as 34ddddcb3eeb):
the schema and messageName options of the protobuf data format, with the new
camel-protobuf-schema module.
_Claude Code on behalf of Claus Ibsen_
> camel-protobuf - a schema option with the .proto text, to read and write a
> DynamicMessage without generated classes (new camel-protobuf-schema)
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27329
> URL: https://issues.apache.org/jira/browse/CAMEL-27329
> Project: Camel
> Issue Type: New Feature
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> The protobuf data format (library GoogleProtobuf) needs {{instanceClass}}, a
> class generated by protoc. A low-code route, such as a YAML route or a
> Kamelet, has no generated classes, so it cannot turn JSON into a Protobuf
> message or read Protobuf binary data.
> This matters most for Kafka with a schema registry: with
> {{schemaRegistryVendor}} and {{schemaFormat=protobuf}}, the Confluent
> {{KafkaProtobufSerializer}} and the Apicurio {{ProtobufKafkaSerializer}} need
> a {{com.google.protobuf.Message}} as the record value. Today a YAML route can
> only build one with a script (the Kamelet tests build a {{DynamicMessage}} in
> Groovy from a hand-made {{FileDescriptorProto}}).
> h3. Proposal
> Two new options of the protobuf data format, used instead of
> {{instanceClass}}:
> * {{schema}}: the text of a .proto file, or a resource such as
> {{classpath:order.proto}}, {{file:order.proto}} or an http URL (a Confluent
> registry serves the text of a schema at
> {{/subjects/<subject>/versions/latest/schema}}).
> * {{messageName}}: the message type to use, such as {{Order}}, {{shop.Order}}
> or the nested {{Order.Line}}. By default the first message type of the schema.
> With {{schema}}, the default instance is
> {{DynamicMessage.getDefaultInstance(descriptor)}}, so everything the data
> format does today works on it:
> * unmarshal JSON ({{contentTypeFormat=json}}) or Protobuf binary data into a
> {{DynamicMessage}};
> * marshal a {{DynamicMessage}}, or a {{Map}} (through the existing
> {{ProtobufConverter.toProto}}), to binary data or JSON.
> {code:yaml}
> - unmarshal:
> protobuf:
> contentTypeFormat: json
> schema: "resource:classpath:order.proto"
> messageName: Order
> - to:
> uri: kafka:orders
> parameters:
> schemaRegistryVendor: confluent
> schemaRegistryURL: http://localhost:8081
> schemaFormat: protobuf
> {code}
> h3. Two modules, so camel-protobuf stays light
> protobuf-java cannot parse .proto text, and the only maintained Java parser
> is the one of Square Wire, which brings Kotlin. So:
> * *camel-protobuf* gets the two options and a small SPI,
> {{org.apache.camel.dataformat.protobuf.ProtobufSchemaParser}} ({{Descriptor
> parse(String schema, String messageName)}}). The data format takes a single
> bean of that type from the registry, or else the implementation named in
> {{META-INF/services/org/apache/camel/protobuf-schema-parser}} (the usual
> FactoryFinder lookup, through {{ResolverHelper.resolveService}}). With
> {{schema}} set and no implementation, it fails at startup with: "The schema
> option of the protobuf data format needs a parser of .proto files: add
> camel-protobuf-schema to the classpath (camel:protobuf-schema as a Kamelet
> dependency)". camel-protobuf gets no new dependency.
> * *camel-protobuf-schema* (new, an "other" module like camel-jsoup) depends
> on camel-protobuf and holds {{WireProtobufSchemaParser}}, registered as the
> {{protobuf-schema-parser}} service. It parses and links the .proto text with
> {{com.squareup.wire:wire-schema-jvm}} ({{squareup-wire-version}} 6.4.7,
> already managed in camel-parent, and the same version that the Confluent
> 8.4.1 and Apicurio 3.3.4 serdes use). wire's {{SchemaEncoder}} writes the
> {{FileDescriptorProto}}, and {{FileDescriptor.buildFrom}} makes the
> descriptor. Imports of the well-known types of {{google/protobuf}}
> (timestamp, duration, struct, wrappers, any, empty, field_mask) use the
> descriptors compiled into protobuf-java, so that JsonFormat handles them in
> their JSON form. Other imports fail with a clear message. The code generators
> of wire (javapoet, kotlinpoet) are excluded; the dependencies it adds are
> wire-schema-jvm (490 KB), wire-runtime-jvm (257 KB), okio-jvm (373 KB),
> kotlin-stdlib (1.7 MB) and guava (3 MB).
> Other Kotlin-free parsers were checked and are not usable:
> com.squareup:protoparser (archived in 2018, last release 2015),
> io.protostuff:protostuff-parser (no release since 2022, needs the ANTLR tool,
> Guava and Guice, and gives its own model, not descriptors), and
> com.github.os72:protoc-jar (runs a bundled native protoc, last release 2020).
> A prototype passes unit tests for proto2 and proto3: nested messages, enums,
> maps, oneof, repeated fields, an imported {{google.protobuf.Timestamp}}, JSON
> -> DynamicMessage, Map -> binary -> DynamicMessage, the errors for an unknown
> message name, a syntax error and an unsupported import, the startup error
> without camel-protobuf-schema, and a parser bean from the registry. Built
> this way, a DynamicMessage is written by {{kafka}} with
> {{schemaFormat=protobuf}} through both a Confluent API registry (Redpanda)
> and Apicurio Registry 3.3.4, and read back by the matching deserializers.
> Also needed: the two options in the model ({{ProtobufDataFormat}} in
> camel-core-model, its reifier, the XML and YAML DSL), the docs of the data
> format, the {{@BodyType}} descriptions, and the new module in the BOM,
> camel-allcomponents, the catalog and the "others" docs.
> {{camel:protobuf-schema}} resolves to
> {{org.apache.camel:camel-protobuf-schema}} in Camel JBang and as a Kamelet
> dependency.
> Motivation: the Kamelet catalog (apache/camel-kamelets) can then offer a
> Kamelet that turns JSON into a Protobuf message for {{kafka-sink}} with
> {{schemaFormat}} {{protobuf}}, for YAML users.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)