davsclaus opened a new pull request, #3295:
URL: https://github.com/apache/camel-kamelets/pull/3295

   Part of #3279, item 1: `protobuf` as a schema format of `kafka-source`, 
`kafka-sink` and `kafka-batch-source`. camel-kafka already supports it for 
`confluent`, `apicurio` and `aws-glue` (`KafkaSchemaRegistry`).
   
   **Stacked:** built on #3286, which is built on #3281. This PR's own change 
is the last commit. Merge it after #3281 and #3286.
   
   ### What changed
   - **`schemaFormat` enum:** now `avro`, `json` and `protobuf` on the three 
Kamelets.
   - **The sources turn a Protobuf record into JSON text.** Before, the body 
was the record's `toString()`, which for a Protobuf `DynamicMessage` is the 
Protobuf text format (`id: 7\nitem: "camel"\n`), not JSON. Now `kafka-source` 
and each entry of `kafka-batch-source` get the JSON of the Protobuf JSON 
mapping, compact, such as `{"id":7,"item":"camel"}`. This uses `JsonFormat` of 
`protobuf-java-util`, which the Confluent and Apicurio Protobuf serdes bring.
   - **The sink** needs a Protobuf message object as the body, such as a 
`DynamicMessage` or an object of a generated class. JSON text fails with 
`SerializationException: Can't convert value of class java.lang.String to class 
...KafkaProtobufSerializer`. The descriptions say this.
   - **Descriptions and partial pages:** they say which vendors support 
`protobuf`, what the body is on each side, and what the JSON mapping does: 
field names in lowerCamelCase (`orderNo` for `order_no`), 64-bit numbers as 
text, and fields with their default value (0, empty text, false) left out. I 
checked each of these with `camel run`.
   
   ### No new dependency: why
   The first version used `marshal: protobuf` with `contentTypeFormat: json` 
and added `camel:protobuf`. That broke Apicurio on Camel JBang:
   - `camel-protobuf` puts `protobuf-java` 4.35.1 (Camel's version) on the 
classpath when the Kamelet loads, before the serde.
   - The serde `apicurio-registry-protobuf-serde-kafka` 3.3.4 has code 
generated with 4.36.2. It fails with `ProtobufRuntimeVersionException: gencode 
4.36.2, runtime 4.35.1`.
   - The consumer then stops reading without logging an error.
   
   So the template calls 
`${bean:type:com.google.protobuf.util.JsonFormat?method=printer().omittingInsignificantWhitespace().print(${body})}`.
 This expression is only evaluated when a Protobuf record is read, so it uses 
the `protobuf-java-util` that the serde brought. A Kamelet without `protobuf` 
loads no Protobuf jar; I checked this with `--verbose`.
   
   Like `avro` and `json`, the serde jars are not listed in 
`spec.dependencies`. camel-kafka names the class, and Camel JBang downloads 
`io.confluent:kafka-protobuf-serializer` or 
`io.apicurio:apicurio-registry-protobuf-serde-kafka` from its known-libraries 
list.
   
   ### Tests
   New Citrus tests in `kafka/`. The Protobuf message is built in a Groovy step 
from a descriptor made in code, so the tests need no `protoc` and no vendor 
class.
   - `kafka-schema-registry-protobuf-route`: `kafka-sink` and then 
`kafka-source` through Redpanda (`confluent`). The body is 
`{"id":1,"item":"camel"}` as a `String`, and the registry has the subject with 
`schemaType` `PROTOBUF`.
   - `kafka-schema-registry-apicurio-protobuf-route`: the same route through 
Apicurio Registry 3.3.4 (`apicurio/apicurio-registry:3.3.4`, which pulls from 
mirror.gcr.io). The artifact `<topic>-value` has `artifactType` `PROTOBUF`.
   - `kafka-batch-schema-registry-protobuf-route`: two records read as one 
batch. Each entry is JSON text.
   
   `./mvnw -o verify -pl tests/camel-kamelets-itest -Denable.integration.tests 
-Dit.test='KafkaIT#kafka'`: **12 of 12 pass**, the new tests and the existing 
Kafka tests. `./mvnw clean install` from the root passes, run twice.
   
   ### Not covered
   - **`aws-glue` with `protobuf`** is not tested: there is no emulator. 
camel-kafka sets `dataFormat=PROTOBUF` for it.
   - **#3279 item 4** (`apicurio` with `json`) is still open, so this PR does 
not close #3279.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to