davsclaus opened a new pull request, #3298:
URL: https://github.com/apache/camel-kamelets/pull/3298

   Batch 5 of #3117 (databases, search, NoSQL): a default data type, `json` 
(`application-json`), on 10 sources (`out`) and 4 sinks (`in`).
   
   > **Draft:** the source defaults need a Camel 4.24.0-SNAPSHOT with 
CAMEL-27258 (the output type of a source Kamelet is applied before the 
consuming route), and the `mongodb-sink` default needs CAMEL-27259 
(camel-mongodb insert of a `byte[]` body). Both were merged in Camel main on 
2026-11-19, but the published snapshot that CI uses predates them. With that 
snapshot, the source tests still get the stream and `MongoIT` fails on the 
`mongodb-sink` insert. Rerun CI once a snapshot with both is published.
   
   ### What changes
   
   | Kamelet | in | out | what changes (before → after) |
   |---|---|---|---|
   | `mariadb-source`, `mysql-source`, `oracle-database-source`, 
`postgresql-source`, `sqlserver-source` | n/a | `json` | body: JSON text as a 
stream (`InputStreamCache`) → JSON text in bytes (`byte[]`). `Content-Type: 
application/json` as before. A JSON array (`useIterator` false) is kept as it 
is. |
   | `aws-redshift-source`, `databricks-source`, `snowflake-source` | n/a | 
`json` | as the SQL sources (same template; no Citrus test) |
   | `mongodb-source` | n/a | `json` | body: stream → `byte[]`; `Content-Type: 
application/json` as before (now in the description) |
   | `solr-source` | n/a | `json` | body: stream → `byte[]`; `Content-Type: 
application/json` as before |
   | `aws-redshift-sink`, `databricks-sink`, `snowflake-sink` | `json` | n/a | 
as the SQL sinks of #3276: the body becomes JSON text in bytes before the 
template parses it; a `Map` or `List` body is now written as JSON first; 
afterwards `Content-Type` is `application/json` (before, the incoming one was 
kept). No Citrus test: the template is the SQL one. |
   | `mongodb-sink` | `json` | n/a | the body becomes JSON text in bytes, and 
`Content-Type: application/json`, for `insert`, `save` and `update`; a `Map` 
(or a `List` for `update`) is now written as JSON first. The stored document is 
the same. Afterwards `Content-Type` is `application/json`. |
   
   The Kamelets do the same as before in what they store or send: each source 
already marshalled its result to JSON, and each sink already parsed JSON text. 
The change is the Java class of the body that the next step sees (sources), and 
the `Content-Type` and the accepted body kinds (sinks).
   
   No row was dropped. Not in this PR: `mongodb-changes-stream-source` (waits 
for C2, and #3283 changes it), and the rows the proposal marks "none" or "after 
a Camel change".
   
   ### Risk to review
   `mongodb-sink` with a `Map` or `Document` body is not documented input (the 
description asks for JSON text), but the template accepted it. With the `json` 
default, such a body goes through Jackson first: a `camel run` of the 
transformer on `{_id: ObjectId(...), when: Date(0), qty: 2}` gave 
`{"_id":{"date":"2024-05-24T14:18:26.000Z","timestamp":1716560306},"when":"1970-01-01T00:00:00.000Z","qty":2}`.
 JSON text in the documented form (`$oid`, `$date`) is unchanged. I kept the 
default, as for the SQL sinks of #3276; drop it from `mongodb-sink` if that 
case matters.
   
   ### Descriptions
   - Sources: the `Output:` paragraph says "JSON text in bytes"; 
`mongodb-source` now also says `Content-Type` is `application/json`.
   - Sinks: the `Output:` paragraph says that `Content-Type` is set to 
`application/json`.
   
   ### Tests
   - `DatabaseIT` (mariadb, mysql, oracle, sqlserver sources), `SearchIT` 
(solr-source): expect `X-Body-Type` `[B` instead of `InputStreamCache`.
   - `PostgresIT`: gets an HTTP server; `postgresql-source` now records the 
body type, the `Content-Type` and the row (it only checked a log line).
   - `MongoIT`:
     - `mongodb-source` records the body type, the `Content-Type` and the 
document (it only checked a log line).
     - `mongodb-sink` now checks the stored document after `insert`, `save` and 
`update` of the same document, and what each leaves on the message (body type, 
`Content-Type`, `CamelMongoOid`, matched and changed counts). Before, it 
checked a log line.
   
   ### Overlap
   - **#3294** (descriptions) changes the same `Output:` sentences of 
`mariadb-`, `mysql-`, `oracle-database-`, `sqlserver-` and `solr-source` (it 
adds "(a stream)"), and the `solr-source` test route. Whichever merges second 
needs a rebase. With this PR the body is bytes, not a stream, so "(a stream)" 
should become "in bytes".
   - **#3283** changes `mongodb-changes-stream-source`, which this PR does not 
touch.
   
   ### Upgrade note (for #3190)
   ```
   * The SQL, MongoDB and Solr sources now give their JSON as bytes: 
`mariadb-source`, `mysql-source`, `oracle-database-source`, 
`postgresql-source`, `sqlserver-source`, `aws-redshift-source`, 
`databricks-source`, `snowflake-source`, `mongodb-source` and `solr-source` 
have an output data type of `json`. The body the next step sees is a `byte[]` 
holding the same JSON text, where it was a stream before. `Content-Type` is 
`application/json`, as before.
   * `aws-redshift-sink`, `databricks-sink`, `snowflake-sink` and 
`mongodb-sink` have an input data type of `json`, as the other SQL sinks have 
since 4.24. They turn their input into JSON bytes and set `Content-Type: 
application/json`, which stays on the message afterwards. They now also accept 
a `Map` (and a `List`), which is written as JSON first. For `mongodb-sink`, a 
`Map` or a MongoDB `Document` body that holds an `ObjectId` or a date is 
therefore stored differently: an `ObjectId` becomes an object of its time 
(`{"date":...,"timestamp":...}`) and a date becomes ISO text. Send such 
documents as JSON text, with `{"$oid":"..."}` and `{"$date":"..."}`.
   ```
   
   ### Test results
   - **Citrus, local** (Camel main with CAMEL-27258 in `~/.m2`):
     - `DatabaseIT`, `PostgresIT`, `SearchIT`: 14 of 14 pass.
     - `MongoIT`: 3 of 3 pass with a camel-mongodb that has CAMEL-27259. The 
camel-mongodb jar in `~/.m2` predates CAMEL-27259; with it, the `mongodb-sink` 
insert fails with `ClassCastException: [B cannot be cast to 
[Ljava.lang.Object;`, as expected. To check the fix without installing Camel 
into `~/.m2`, `MongoDbProducer` from Camel main was compiled into a copy of 
that jar, which was passed to `camel run` only for this local run (not 
committed).
   - **Build:** `./mvnw clean install` from the root passes, run twice.
   - Redshift, Databricks and Snowflake have no Citrus test; their templates 
are the SQL ones that `DatabaseIT` and `PostgresIT` cover.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to