davsclaus opened a new pull request, #3308:
URL: https://github.com/apache/camel-kamelets/pull/3308

   Part of #3117, batch 9a. Declares the new no-op data type 
`application-x-unchanged` (CAMEL-27269) as the default data type of the 20 
actions and 5 sinks that leave the body as it is or don't read it. Its 
transformer does nothing: the body, its Java type and the headers stay as they 
are, and no `Content-Type` is set. What it adds is the contract: the YAML 
validator (`camel validate`, CAMEL-25928) then knows that the body after the 
Kamelet is the body before it. Camel still runs the default on every message, 
so each Kamelet was checked with `camel run` against `main` and with the Citrus 
tests.
   
   ## What changes
   
   | Kamelet | in | out | Checked: the body is |
   |---|---|---|---|
   | `delay-action`, `throttle-action` | `unchanged` | `unchanged` | not read |
   | `drop-header-action`, `drop-headers-action`, `insert-header-action` | 
`unchanged` | `unchanged` | not read (`insert-header-action`: only when `value` 
reads it) |
   | `has-header-filter-action`, `header-matches-filter-action`, 
`topic-name-matches-filter-action` | `unchanged` | `unchanged` | not read |
   | `is-tombstone-filter-action`, `simple-filter-action` | `unchanged` | 
`unchanged` | compared, not changed |
   | `predicate-filter-action` | `unchanged` | `unchanged` | read by JSONPath, 
not changed |
   | `kafka-manual-commit-action` | `unchanged` | `unchanged` | not read 
(`ManualCommit` reads a header) |
   | `kafka-batch-manual-commit-action` | `unchanged` | `unchanged` | read as a 
List for the header of its last entry, not changed (`BatchManualCommit`) |
   | `regex-router-action`, `timestamp-router-action`, `set-kafka-key-action` | 
`unchanged` | `unchanged` | not read (`RegexRouter`, `TimestampRouter` set 
headers only) |
   | `log-action` | `unchanged` | `unchanged` | logged, not changed |
   | `resolve-pojo-schema-action` | `unchanged` | `unchanged` | not changed: 
the schema resolvers set exchange properties only |
   | `json-schema-validator-action` | `unchanged` | `unchanged` | read, not 
changed (see Good to know) |
   | `pqc-signature-action` (`sign`, `verify`) | `unchanged` | `unchanged` | 
read, not changed: `PQCProducer` sets the signature headers only |
   | `log-sink` | `unchanged` | - | logged, not changed |
   | `aws-cloudwatch-sink` | `unchanged` | - | not changed |
   | `aws-ec2-sink`, `aws-timestream-query-sink`, `aws-bedrock-agent-sink` | 
`unchanged` | - | not used (no `pojoRequest`); the body afterwards is the AWS 
answer, as before |
   
   The Kamelets that already had a `dataTypes` block keep their headers; 
`default` and `types` were added next to them. No description changed: nothing 
visible changes.
   
   `documenting.adoc` said that a Kamelet that transforms nothing declares the 
`headers` block alone. It now describes the `application-x-unchanged` data type 
too, with an example, and `development.adoc` points to it.
   
   ### What is visible
   
   Only the data type that Camel records on the message. After one of these 
Kamelets it is `application-x-unchanged`, so the data type an earlier Kamelet 
set, such as `application-json`, is forgotten. A Kamelet after it with an `in` 
default then runs its transformer again, where Camel used to skip it because 
the message already had that type. For the `json`, `text` and `binary` 
transformers this gives the same result. Checked with `camel run`, using test 
Kamelets with an `in` default of `json`, `text` or `binary`:
   
   - `json-serialize-action` → `insert-header-action` → `in: json`:
     - before: the message reaches the third Kamelet as `application-json`;
     - now: it reaches it as `application-x-unchanged`;
     - the third Kamelet gives `byte[]` `{"foo":"bar"}`, `Content-Type: 
application/json`, before and after.
   - The text body `hello` gives `"hello"`, not encoded twice.
   - The same holds with `predicate-filter-action` or `insert-header-action` → 
`log-action` in the middle, and with `in: text` (a String, `text/plain`) and 
`in: binary` (`byte[]`, `application/octet-stream`) at the end.
   - `data-type-action` → `resolve-pojo-schema-action` → `data-type-action` 
(the Avro and Protobuf tests) still finds its transformer.
   
   A message that a filter drops never reaches the end of the Kamelet, so it 
gets no output type (CAMEL-27290).
   
   ## How it was checked
   
   `camel run`, fresh JVM, the Kamelets of `main` and then of this branch:
   
   - **Actions and `log-sink`:** each one with a `String`, a `Map` and a 
`byte[]` body, via the Kamelet EIP (`- kamelet:`) and via `to: kamelet:`. The 
route logged the body class, the body, `Content-Type` and the headers the 
Kamelet sets. That is 126 cases.
   - **Result:** before and after are the same in every case except the 
message's data type: `java:<class>` → `application-x-unchanged`.
   - **Failures:** the same 4 cases fail before and after. A `Map` body fails 
in `json-schema-validator-action` (as its description says) and in 
`pqc-signature-action` (`InvalidPayloadException`, a `Map` has no bytes to 
sign).
   - **AWS sinks:** against a stub of the AWS API. Before and after, the body 
afterwards is:
     - `aws-cloudwatch-sink`: the `String` it came with;
     - `aws-ec2-sink`: a `DescribeInstanceStatusResponse`;
     - `aws-timestream-query-sink`: a `QueryResponse`;
     - `aws-bedrock-agent-sink`: the job ID;
     - no `Content-Type` in all four.
   
   ## Not done
   
   - `message-timestamp-router-action` and `value-to-key-action` read the body 
as JSON; they get `json` defaults in #3299.
   - No sink or action in the list was left out. All 25 leave the body and its 
class as they are, by reading the template and the Camel code (read-only) and 
by running them.
   - Sources get no `unchanged` out type: it says nothing about what a source 
gives.
   
   ## Good to know
   
   With stream caching turned off (it is on by default), 
`json-schema-validator-action` replaces an `InputStream` body with a 
`StreamCache` so that it can be read again. A plain stream is also used up by 
`pqc-signature-action`. Both behave the same before and after this change.
   
   ## Tests
   
   The tests now check the body class and `Content-Type` after the Kamelet 
(headers `X-Body-Type` and `X-Content-Type`, checked by Citrus `http: 
receiveRequest`). That was added to:
   
   - `delay`, `throttle`, the three header actions and the four filters in 
`filter/`;
   - `log-sink`: it only checked the log before. It now also sends the message 
on, and its timer stops after one message;
   - `regex-router-action`, `timestamp-router-action` and 
`topic-name-matches-filter-action`: they keep their log check, and now also 
send the message on with the new topic in `X-Override-Topic`;
   - `kafka-manual-commit-action`: `Content-Type` was added, the body class was 
already checked;
   - the batch route: the body class and `Content-Type` after 
`kafka-batch-manual-commit-action`.
   
   `json-schema-validator-action`, `pqc-signature-action`, 
`is-tombstone-filter-action`, `set-kafka-key-action` and the 4 AWS sinks 
already checked both. The expected values are those of `main`, as `camel run` 
showed.
   
   Results (local, Camel 4.24.0-SNAPSHOT from `apache/camel` main of today):
   - 
`CommonIT#delay+throttle+header+filter+log+json+kafkaRouter+pqc+transformation+crypto`:
 39 of 39 pass.
   - `AwsIT#awsCloudWatch+awsEc2+awsTimestream+awsBedrock`: 6 of 6 pass. The 
EC2 and CloudWatch tests run against the Floci container.
   - `KafkaIT`: 6 of 6 pass, including `kafka-manual-commit-action`, the batch 
route with `kafka-batch-manual-commit-action`, and `kafka-router-route` 
(`timestamp-router-action` → `set-body-action` → `log-action` → `kafka-sink`).
   - `AvroIT` and `ProtobufIT`: 4 of 4 pass. These run 
`resolve-pojo-schema-action` and `log-action` between `data-type-action` steps.
   - `./mvnw clean install` from the root, twice: green.
   
   ## Overlap
   
   - #3287 adds the `verified` label to the metadata of many of the same 
Kamelets. This PR changes `spec.dataTypes`, a different place in the file, so 
there should be no conflict.
   - #3306 changes the Kafka endpoints of 
`kafka-manual-commit-action-route.citrus.it.yaml` and 
`kafka-batch-source-route.citrus.it.yaml`. This PR adds header checks lower 
down in the same files, so whichever merges second may need a small rebase.
   - #3285 changes the options of `aws-timestream-query-sink`. This PR only 
adds `default` and `types` under `dataTypes.in`, so there should be no conflict.
   
   ## Needs
   
   The Camel 4.24.0-SNAPSHOT of tonight, which has:
   - CAMEL-27269: the `application-x-unchanged` data type;
   - CAMEL-27289: the transformer registry no longer lets the `from` type 
shadow the next Kamelet's `in` transformer;
   - CAMEL-27290: the `out` type of a Kamelet is applied at `kamelet:sink`.
   
   The published snapshot does not have them yet. Without CAMEL-27269 Camel 
finds no transformer for `application-x-unchanged`. I checked this with 
`delay-action` on Camel 4.23.0: for a `String` body Camel then does nothing, 
but the result for other bodies and in chains was not checked. So this PR is a 
draft until CI runs on tonight's snapshot.
   
   ## Upgrade note for #3190 (Data types section)
   
   - **The no-op data type on the Kamelets that leave the body as it is (#PR):**
     - These Kamelets now declare the data type `application-x-unchanged` as 
their default:
       - the actions `delay-action`, `throttle-action`, `drop-header-action`, 
`drop-headers-action`, `insert-header-action`, `has-header-filter-action`, 
`header-matches-filter-action`, `is-tombstone-filter-action`, 
`predicate-filter-action`, `simple-filter-action`, 
`topic-name-matches-filter-action`, `kafka-manual-commit-action`, 
`kafka-batch-manual-commit-action`, `regex-router-action`, 
`set-kafka-key-action`, `timestamp-router-action`, `log-action`, 
`resolve-pojo-schema-action`, `json-schema-validator-action` and 
`pqc-signature-action`, for `in` and `out`;
       - the sinks `log-sink`, `aws-cloudwatch-sink`, `aws-ec2-sink`, 
`aws-timestream-query-sink` and `aws-bedrock-agent-sink`, for `in`.
     - The body, its type and the headers stay as they are. `camel validate` 
now follows the body through them.
     - The data type of the message is now `application-x-unchanged` after 
them. A Kamelet after one of them with an input data type therefore runs its 
transformer, also when an earlier Kamelet had already given the message that 
type. For the `json`, `text` and `binary` data types the result is the same.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to