davsclaus opened a new pull request, #3301:
URL: https://github.com/apache/camel-kamelets/pull/3301
Batch 2 (Azure and Google Cloud) of #3117: a default data type on the Azure
and Google Cloud Kamelets whose body is fixed. Camel runs the transformer of a
default data type on every message, so each default changes the body class or
the `Content-Type`. The table shows what changes. It was checked with the
Citrus tests and with `camel run`, on a local Camel main build that has
CAMEL-27258.
> **Draft:** the `azure-servicebus-source` and `google-pubsub-source` tests
need CAMEL-27258, which applies a source's output type before the consuming
route. That fix was merged today, after the published Camel 4.24.0-SNAPSHOT
that CI uses was built. This PR waits for a newer snapshot, like #3297.
## What changes
| Kamelet | in | out | What changes (before → after) |
|---|---|---|---|
| `azure-servicebus-source` | - | `binary` | `byte[]`, no `Content-Type` →
`byte[]`, `Content-Type: application/octet-stream`. The content type that the
sender set stays in `CamelAzureServiceBusContentType`. |
| `google-pubsub-source` | - | `binary` | `byte[]`, no `Content-Type` →
`byte[]`, `Content-Type: application/octet-stream`. A `Content-Type` attribute
that the publisher set was copied to the `Content-Type` header; now it is only
in `CamelGooglePubsubAttributes`. |
| `azure-cosmosdb-sink` | `json` | - | JSON text `String` → JSON `byte[]`,
before the template's `unmarshal`. A Map is now written as JSON and stored (it
failed before). Plain text and a List still fail, now in Cosmos DB instead of
in `unmarshal`. A number fails. After the call: the created item as a
`LinkedHashMap` as before, `Content-Type` → `application/json`. |
| `google-bigquery-sink` | `json` | - | as `azure-cosmosdb-sink`: a Map or a
List of Maps is now written as JSON and inserted (they failed before). Plain
text still fails, now in BigQuery. After the call: the parsed JSON
(`LinkedHashMap` or `ArrayList`) as before, `Content-Type` →
`application/json`. |
| `azure-storage-queue-sink` | `text` | - | bytes, a stream, a Map or a
number → `String` at the start of the Kamelet (read as UTF-8). Azure receives
the same text as before, since the producer read the body as a `String` anyway.
After the call: `Boolean` `true` as before, `Content-Type` → `text/plain`. |
The `Input:`/`Output:` paragraphs of these Kamelets now say this.
## Not done
- **`google-vertexai-sink`: no default.** The proposal had `text` here. A
`camel run` with the default showed it breaks valid input of three operations
the Kamelet offers, because the transformer runs at the start of the Kamelet
for every operation:
- `generateEmbeddings`: a List of texts (one vector per text) became the
single text `[first text, second text]`.
- `rawPredict`: a Map, which Camel writes as the JSON request, became
`{id=order-1, customer=citrus}`, which is not JSON.
- `generateMultimodal`: a file body, which Camel sends as the media,
failed to convert to text.
The YAML comment now gives this reason.
- The other Azure and Google Kamelets of the proposal wait on a Camel change
(the blob, data lake, files and storage sinks wait on C3), keep their default,
or get none. The Pub/Sub, Functions and Service Bus sinks keep no default:
Pub/Sub and Functions forward `Content-Type` (D2), and Service Bus and Event
Hubs take a List.
## Tests
- `AzureIT` and `GoogleCloudIT`, run locally with Camel main + CAMEL-27258
in `~/.m2`: the five tests of this PR pass (`azure-servicebus-source`,
`azure-cosmosdb-sink`, `azure-storage-queue-sink`, `google-pubsub-source`,
`google-bigquery-sink`), and so do the other sink and event-based source tests
(15 of 21).
- The other 6 fail, on files this PR does not touch: the tests of the
sources that already had a default, which CAMEL-27258 now applies before the
consuming route. They expect what the CI snapshot gives today:
- `google-storage-source`, `azure-storage-blob-source`,
`azure-storage-datalake-source`, `azure-eventhubs-source` (`binary`):
`X-Content-Type` is now `ct=application/octet-stream`;
- `azure-cosmosdb-source` (`json`): `X-Body-Type` is now `[B`;
- `azure-storage-queue-source` (`text`): `X-Body-Type` is now
`java.lang.String`.
They will fail on `main` too once CI has a snapshot with CAMEL-27258, so
they need their own follow-up (the CAMEL-27258 bullet of #3190 lists these
Kamelets).
- The expected `X-Body-Type`/`X-Content-Type` values are updated. All five
routes already recorded them.
- The Service Bus and Pub/Sub source routes remove `Content-Type` before
they send the body to the Citrus server, because Citrus reads an
`application/octet-stream` body as a byte array.
- The Pub/Sub source test now publishes a `Content-Type: text/plain`
attribute and checks that the header is `application/octet-stream` while
`CamelGooglePubsubAttributes` keeps the attribute.
- `camel run` with stand-ins (each Kamelet with its component call replaced
by a log of the body), before and after, with JSON text, JSON bytes, a Map, a
List, plain text, a stream and a number as the body. The results are in the
table.
- `azure-cosmosdb-sink` by hand against the Cosmos DB emulator: JSON text
and a Map are stored (201); plain text fails with `The body of the createItem
operation is not valid JSON`, a List with `must be one item`.
- `google-bigquery-sink` by hand against the Floci GCP emulator: a Map and a
List of Maps are inserted (3 rows in the table); plain text fails with `Cannot
handle body type java.lang.String`.
- `./mvnw clean install` from the root passes, run twice.
## Overlap with open PRs
- None of the six Kamelets or their tests are touched by #3294, #3285 or
#3297. Those PRs change the generated `kamelets-index.json`, the SBOM or the
copies under `library/camel-kamelets`, so the PR that merges second regenerates
them.
## Upgrade note for #3190
```markdown
- **Default data types on the Azure and Google Cloud Kamelets (#PR_NUMBER):**
- `azure-servicebus-source` and `google-pubsub-source` (`out: binary`) set
`Content-Type: application/octet-stream` on the bytes. With
`google-pubsub-source`, a `Content-Type` attribute set by the publisher no
longer becomes the `Content-Type` header: read it from
`CamelGooglePubsubAttributes`. With `azure-servicebus-source`, the content type
of the sender is in `CamelAzureServiceBusContentType`, as before.
- `azure-cosmosdb-sink` and `google-bigquery-sink` (`in: json`) now also
take a `Map` (and BigQuery a `List` of Maps), which is written as JSON.
`Content-Type: application/json` stays on the message after the sink. A number
fails at the start of the Kamelet.
- `azure-storage-queue-sink` (`in: text`) reads the body as UTF-8 text at
the start of the Kamelet and sets `Content-Type: text/plain`, which stays on
the message after the sink. The queue receives the same text as before.
```
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]