davsclaus opened a new pull request, #3308:
URL: https://github.com/apache/camel-kamelets/pull/3308
Part of #3117, batch 9a. Declares the new no-op data type
`application-x-unchanged` (CAMEL-27269) as the default data type of the 20
actions and 5 sinks that leave the body as it is or don't read it. Its
transformer does nothing: the body, its Java type and the headers stay as they
are, and no `Content-Type` is set. What it adds is the contract: the YAML
validator (`camel validate`, CAMEL-25928) then knows that the body after the
Kamelet is the body before it. Camel still runs the default on every message,
so each Kamelet was checked with `camel run` against `main` and with the Citrus
tests.
## What changes
| Kamelet | in | out | Checked: the body is |
|---|---|---|---|
| `delay-action`, `throttle-action` | `unchanged` | `unchanged` | not read |
| `drop-header-action`, `drop-headers-action`, `insert-header-action` |
`unchanged` | `unchanged` | not read (`insert-header-action`: only when `value`
reads it) |
| `has-header-filter-action`, `header-matches-filter-action`,
`topic-name-matches-filter-action` | `unchanged` | `unchanged` | not read |
| `is-tombstone-filter-action`, `simple-filter-action` | `unchanged` |
`unchanged` | compared, not changed |
| `predicate-filter-action` | `unchanged` | `unchanged` | read by JSONPath,
not changed |
| `kafka-manual-commit-action` | `unchanged` | `unchanged` | not read
(`ManualCommit` reads a header) |
| `kafka-batch-manual-commit-action` | `unchanged` | `unchanged` | read as a
List for the header of its last entry, not changed (`BatchManualCommit`) |
| `regex-router-action`, `timestamp-router-action`, `set-kafka-key-action` |
`unchanged` | `unchanged` | not read (`RegexRouter`, `TimestampRouter` set
headers only) |
| `log-action` | `unchanged` | `unchanged` | logged, not changed |
| `resolve-pojo-schema-action` | `unchanged` | `unchanged` | not changed:
the schema resolvers set exchange properties only |
| `json-schema-validator-action` | `unchanged` | `unchanged` | read, not
changed (see Good to know) |
| `pqc-signature-action` (`sign`, `verify`) | `unchanged` | `unchanged` |
read, not changed: `PQCProducer` sets the signature headers only |
| `log-sink` | `unchanged` | - | logged, not changed |
| `aws-cloudwatch-sink` | `unchanged` | - | not changed |
| `aws-ec2-sink`, `aws-timestream-query-sink`, `aws-bedrock-agent-sink` |
`unchanged` | - | not used (no `pojoRequest`); the body afterwards is the AWS
answer, as before |
The Kamelets that already had a `dataTypes` block keep their headers;
`default` and `types` were added next to them. No description changed: nothing
visible changes.
`documenting.adoc` said that a Kamelet that transforms nothing declares the
`headers` block alone. It now describes the `application-x-unchanged` data type
too, with an example, and `development.adoc` points to it.
### What is visible
Only the data type that Camel records on the message. After one of these
Kamelets it is `application-x-unchanged`, so the data type an earlier Kamelet
set, such as `application-json`, is forgotten. A Kamelet after it with an `in`
default then runs its transformer again, where Camel used to skip it because
the message already had that type. For the `json`, `text` and `binary`
transformers this gives the same result. Checked with `camel run`, using test
Kamelets with an `in` default of `json`, `text` or `binary`:
- `json-serialize-action` → `insert-header-action` → `in: json`:
- before: the message reaches the third Kamelet as `application-json`;
- now: it reaches it as `application-x-unchanged`;
- the third Kamelet gives `byte[]` `{"foo":"bar"}`, `Content-Type:
application/json`, before and after.
- The text body `hello` gives `"hello"`, not encoded twice.
- The same holds with `predicate-filter-action` or `insert-header-action` →
`log-action` in the middle, and with `in: text` (a String, `text/plain`) and
`in: binary` (`byte[]`, `application/octet-stream`) at the end.
- `data-type-action` → `resolve-pojo-schema-action` → `data-type-action`
(the Avro and Protobuf tests) still finds its transformer.
A message that a filter drops never reaches the end of the Kamelet, so it
gets no output type (CAMEL-27290).
## How it was checked
`camel run`, fresh JVM, the Kamelets of `main` and then of this branch:
- **Actions and `log-sink`:** each one with a `String`, a `Map` and a
`byte[]` body, via the Kamelet EIP (`- kamelet:`) and via `to: kamelet:`. The
route logged the body class, the body, `Content-Type` and the headers the
Kamelet sets. That is 126 cases.
- **Result:** before and after are the same in every case except the
message's data type: `java:<class>` → `application-x-unchanged`.
- **Failures:** the same 4 cases fail before and after. A `Map` body fails
in `json-schema-validator-action` (as its description says) and in
`pqc-signature-action` (`InvalidPayloadException`, a `Map` has no bytes to
sign).
- **AWS sinks:** against a stub of the AWS API. Before and after, the body
afterwards is:
- `aws-cloudwatch-sink`: the `String` it came with;
- `aws-ec2-sink`: a `DescribeInstanceStatusResponse`;
- `aws-timestream-query-sink`: a `QueryResponse`;
- `aws-bedrock-agent-sink`: the job ID;
- no `Content-Type` in all four.
## Not done
- `message-timestamp-router-action` and `value-to-key-action` read the body
as JSON; they get `json` defaults in #3299.
- No sink or action in the list was left out. All 25 leave the body and its
class as they are, by reading the template and the Camel code (read-only) and
by running them.
- Sources get no `unchanged` out type: it says nothing about what a source
gives.
## Good to know
With stream caching turned off (it is on by default),
`json-schema-validator-action` replaces an `InputStream` body with a
`StreamCache` so that it can be read again. A plain stream is also used up by
`pqc-signature-action`. Both behave the same before and after this change.
## Tests
The tests now check the body class and `Content-Type` after the Kamelet
(headers `X-Body-Type` and `X-Content-Type`, checked by Citrus `http:
receiveRequest`). That was added to:
- `delay`, `throttle`, the three header actions and the four filters in
`filter/`;
- `log-sink`: it only checked the log before. It now also sends the message
on, and its timer stops after one message;
- `regex-router-action`, `timestamp-router-action` and
`topic-name-matches-filter-action`: they keep their log check, and now also
send the message on with the new topic in `X-Override-Topic`;
- `kafka-manual-commit-action`: `Content-Type` was added, the body class was
already checked;
- the batch route: the body class and `Content-Type` after
`kafka-batch-manual-commit-action`.
`json-schema-validator-action`, `pqc-signature-action`,
`is-tombstone-filter-action`, `set-kafka-key-action` and the 4 AWS sinks
already checked both. The expected values are those of `main`, as `camel run`
showed.
Results (local, Camel 4.24.0-SNAPSHOT from `apache/camel` main of today):
-
`CommonIT#delay+throttle+header+filter+log+json+kafkaRouter+pqc+transformation+crypto`:
39 of 39 pass.
- `AwsIT#awsCloudWatch+awsEc2+awsTimestream+awsBedrock`: 6 of 6 pass. The
EC2 and CloudWatch tests run against the Floci container.
- `KafkaIT`: 6 of 6 pass, including `kafka-manual-commit-action`, the batch
route with `kafka-batch-manual-commit-action`, and `kafka-router-route`
(`timestamp-router-action` → `set-body-action` → `log-action` → `kafka-sink`).
- `AvroIT` and `ProtobufIT`: 4 of 4 pass. These run
`resolve-pojo-schema-action` and `log-action` between `data-type-action` steps.
- `./mvnw clean install` from the root, twice: green.
## Overlap
- #3287 adds the `verified` label to the metadata of many of the same
Kamelets. This PR changes `spec.dataTypes`, a different place in the file, so
there should be no conflict.
- #3306 changes the Kafka endpoints of
`kafka-manual-commit-action-route.citrus.it.yaml` and
`kafka-batch-source-route.citrus.it.yaml`. This PR adds header checks lower
down in the same files, so whichever merges second may need a small rebase.
- #3285 changes the options of `aws-timestream-query-sink`. This PR only
adds `default` and `types` under `dataTypes.in`, so there should be no conflict.
## Needs
The Camel 4.24.0-SNAPSHOT of tonight, which has:
- CAMEL-27269: the `application-x-unchanged` data type;
- CAMEL-27289: the transformer registry no longer lets the `from` type
shadow the next Kamelet's `in` transformer;
- CAMEL-27290: the `out` type of a Kamelet is applied at `kamelet:sink`.
The published snapshot does not have them yet. Without CAMEL-27269 Camel
finds no transformer for `application-x-unchanged`. I checked this with
`delay-action` on Camel 4.23.0: for a `String` body Camel then does nothing,
but the result for other bodies and in chains was not checked. So this PR is a
draft until CI runs on tonight's snapshot.
## Upgrade note for #3190 (Data types section)
- **The no-op data type on the Kamelets that leave the body as it is (#PR):**
- These Kamelets now declare the data type `application-x-unchanged` as
their default:
- the actions `delay-action`, `throttle-action`, `drop-header-action`,
`drop-headers-action`, `insert-header-action`, `has-header-filter-action`,
`header-matches-filter-action`, `is-tombstone-filter-action`,
`predicate-filter-action`, `simple-filter-action`,
`topic-name-matches-filter-action`, `kafka-manual-commit-action`,
`kafka-batch-manual-commit-action`, `regex-router-action`,
`set-kafka-key-action`, `timestamp-router-action`, `log-action`,
`resolve-pojo-schema-action`, `json-schema-validator-action` and
`pqc-signature-action`, for `in` and `out`;
- the sinks `log-sink`, `aws-cloudwatch-sink`, `aws-ec2-sink`,
`aws-timestream-query-sink` and `aws-bedrock-agent-sink`, for `in`.
- The body, its type and the headers stay as they are. `camel validate`
now follows the body through them.
- The data type of the message is now `application-x-unchanged` after
them. A Kamelet after one of them with an input data type therefore runs its
transformer, also when an earlier Kamelet had already given the message that
type. For the `json`, `text` and `binary` data types the result is the same.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]