davsclaus opened a new pull request, #3295:
URL: https://github.com/apache/camel-kamelets/pull/3295
Part of #3279, item 1: `protobuf` as a schema format of `kafka-source`,
`kafka-sink` and `kafka-batch-source`. camel-kafka already supports it for
`confluent`, `apicurio` and `aws-glue` (`KafkaSchemaRegistry`).
**Stacked:** built on #3286, which is built on #3281. This PR's own change
is the last commit. Merge it after #3281 and #3286.
### What changed
- **`schemaFormat` enum:** now `avro`, `json` and `protobuf` on the three
Kamelets.
- **The sources turn a Protobuf record into JSON text.** Before, the body
was the record's `toString()`, which for a Protobuf `DynamicMessage` is the
Protobuf text format (`id: 7\nitem: "camel"\n`), not JSON. Now `kafka-source`
and each entry of `kafka-batch-source` get the JSON of the Protobuf JSON
mapping, compact, such as `{"id":7,"item":"camel"}`. This uses `JsonFormat` of
`protobuf-java-util`, which the Confluent and Apicurio Protobuf serdes bring.
- **The sink** needs a Protobuf message object as the body, such as a
`DynamicMessage` or an object of a generated class. JSON text fails with
`SerializationException: Can't convert value of class java.lang.String to class
...KafkaProtobufSerializer`. The descriptions say this.
- **Descriptions and partial pages:** they say which vendors support
`protobuf`, what the body is on each side, and what the JSON mapping does:
field names in lowerCamelCase (`orderNo` for `order_no`), 64-bit numbers as
text, and fields with their default value (0, empty text, false) left out. I
checked each of these with `camel run`.
### No new dependency: why
The first version used `marshal: protobuf` with `contentTypeFormat: json`
and added `camel:protobuf`. That broke Apicurio on Camel JBang:
- `camel-protobuf` puts `protobuf-java` 4.35.1 (Camel's version) on the
classpath when the Kamelet loads, before the serde.
- The serde `apicurio-registry-protobuf-serde-kafka` 3.3.4 has code
generated with 4.36.2. It fails with `ProtobufRuntimeVersionException: gencode
4.36.2, runtime 4.35.1`.
- The consumer then stops reading without logging an error.
So the template calls
`${bean:type:com.google.protobuf.util.JsonFormat?method=printer().omittingInsignificantWhitespace().print(${body})}`.
This expression is only evaluated when a Protobuf record is read, so it uses
the `protobuf-java-util` that the serde brought. A Kamelet without `protobuf`
loads no Protobuf jar; I checked this with `--verbose`.
Like `avro` and `json`, the serde jars are not listed in
`spec.dependencies`. camel-kafka names the class, and Camel JBang downloads
`io.confluent:kafka-protobuf-serializer` or
`io.apicurio:apicurio-registry-protobuf-serde-kafka` from its known-libraries
list.
### Tests
New Citrus tests in `kafka/`. The Protobuf message is built in a Groovy step
from a descriptor made in code, so the tests need no `protoc` and no vendor
class.
- `kafka-schema-registry-protobuf-route`: `kafka-sink` and then
`kafka-source` through Redpanda (`confluent`). The body is
`{"id":1,"item":"camel"}` as a `String`, and the registry has the subject with
`schemaType` `PROTOBUF`.
- `kafka-schema-registry-apicurio-protobuf-route`: the same route through
Apicurio Registry 3.3.4 (`apicurio/apicurio-registry:3.3.4`, which pulls from
mirror.gcr.io). The artifact `<topic>-value` has `artifactType` `PROTOBUF`.
- `kafka-batch-schema-registry-protobuf-route`: two records read as one
batch. Each entry is JSON text.
`./mvnw -o verify -pl tests/camel-kamelets-itest -Denable.integration.tests
-Dit.test='KafkaIT#kafka'`: **12 of 12 pass**, the new tests and the existing
Kafka tests. `./mvnw clean install` from the root passes, run twice.
### Not covered
- **`aws-glue` with `protobuf`** is not tested: there is no emulator.
camel-kafka sets `dataFormat=PROTOBUF` for it.
- **#3279 item 4** (`apicurio` with `json`) is still open, so this PR does
not close #3279.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]