davsclaus opened a new pull request, #3237:
URL: https://github.com/apache/camel-kamelets/pull/3237

   Part of #3170: `snowflake-source` and `snowflake-sink` can choose the 
warehouse, the schema and the role of the session. 
`mongodb-changes-stream-source` gets no new option: it cannot resume after a 
restart without a bean. Its docs now say that.
   
   ### The options
   
   `snowflake-source` and `snowflake-sink`. They are set on the data source 
bean `dsBean`, next to `databaseName`. The bean is 
`SnowflakeDataSourceFactory.createDataSource()`, as before #3208 and after it.
   
   | Option | Template mapping (bean `dsBean`) | What it does |
   |---|---|---|
   | `warehouse` | `warehouse: "{{?warehouse}}"` | The virtual warehouse that 
runs the queries or statements. |
   | `schema` | `schema: "{{?schema}}"` | The schema of `databaseName` that 
holds the tables. |
   | `role` | `role: "{{?role}}"` | The role to run as, which must be granted 
to the user. |
   
   - **Bean setters, not `properties[...]`:** snowflake-jdbc 4.3.4 has the 
setters `setWarehouse`, `setSchema` and `setRole` on 
`SnowflakeBasicDataSource`. Each one puts its `SFSessionProperty` key into the 
same `properties` map that #3208 fills with the key options. So these map like 
`databaseName` does. #3208 used `properties[...]` only because the key setters 
take two arguments.
   - **What reaches Snowflake:** the driver sends them as the `warehouse`, 
`schemaName` and `roleName` query parameters of `POST 
/session/v1/login-request`, next to `databaseName`.
   - **Optional, no `default`:** with `{{?x}}` unset, the setter is not called. 
The login request and the endpoint URI are then identical to `main`'s; see the 
table below. Snowflake then uses the user's `DEFAULT_WAREHOUSE`, 
`DEFAULT_NAMESPACE` and `DEFAULT_ROLE`.
   - **No `pattern:`:** none of them goes into a URI. The bean gets them as 
they are, and the driver encodes them as query parameters.
   - **Names:** `warehouse`, `schema` and `role` are the driver's names (the 
JDBC connection parameters, and the `SnowflakeDataSource` setters). None of 
them is an option name that the `kamelet:` component takes.
   - **Descriptions:**
     - "Good to know" of both Kamelets names the three options and says that 
the user's defaults apply when they are not set.
     - Each option says what happens when it is not set.
     - Both partial pages have a "Warehouse, Schema and Role" section with an 
example, and a "Connections" section (see below).
     - The statements about Snowflake's defaults and about a statement failing 
with no warehouse come from Snowflake's [JDBC connection 
parameters](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-parameters) 
and user documentation. They were **not** run against Snowflake.
   
   ### Left out
   
   - **Pool settings (`maxTotal`, `maxIdle`, `maxWaitMillis`)**, as #3215 adds 
them to the other SQL Kamelets. There, the bean is a commons-dbcp2 
`BasicDataSource`. Here it is the driver's `SnowflakeBasicDataSource`, which 
does not pool: each `getConnection()` logs in and opens a new Snowflake 
session, and closing the connection ends it. In the tests below, the sink 
logged in once per message (2 messages, 2 logins) and the source once per poll 
(4 polls, 4 logins). The "Good to know" paragraphs and the partial pages now 
say so.
     - A pool would mean a different bean: a commons-dbcp2 `BasicDataSource` 
with the Snowflake driver class, with the key options passed through 
`connectionProperties`. Long-lived pooled sessions also raise 
Snowflake-specific questions, such as session expiry and keep-alive, and that 
needs its own PR and its own tests. `commons-dbcp2` is still in 
`spec.dependencies` from before #3055, but nothing uses it.
   - **Resuming `mongodb-changes-stream-source` after a restart.** 
camel-mongodb 4.24.0-SNAPSHOT (the same in 4.23.0) has three options that look 
related. None of them works in this Kamelet without a bean:
     - `changeStreamTokenRepository`: this is what resumes. After each 
exchange, the consumer saves the change's resume token in a `StateRepository`, 
keyed by route ID and collection, and starts after it on the next start. It 
takes an object, such as a `FileStateRepository`. A Kamelet can declare a bean, 
but it creates the bean every time, and the endpoint must name it every time. 
An optional placeholder drops a URI parameter only when the value starts with 
the placeholder, so 
`#class:...FileStateRepository#fileStateRepository('{{?file}}')` is not dropped 
when unset. The option cannot be made optional with the endpoint unchanged from 
`main`.
     - `changeStreamToken`: a fixed token to start after. It is read on 
**every** start, and it wins over the repository, so each restart sends the 
same changes again (see the table). The Kamelet does not send the tokens as 
text either: the only header that carries one, `CamelOffset`, holds a Java 
object. So a user has no token to set.
     - `persistentTailTracking` (with `persistentId`): only the tailable cursor 
consumer (`mongodb-source`) uses it. With `consumerType=changeStreams`, it does 
nothing (see the table).
   
     Instead, the "Good to know" paragraph and a new "Restarts" section of the 
partial page say that the changes made while the Kamelet is stopped are never 
sent. They also say that the collection must exist when the Kamelet starts; 
otherwise it fails with `Collection orders does not exist on the database and 
createCollection is false`, which happened in a test. Supporting resume would 
need a Camel change first, such as a camel-mongodb option that takes a file 
path or that stores the tokens in a MongoDB collection.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT, `camel run ... --local-kamelet-dir`.
   
   **Snowflake.** There is no local Snowflake, and **nothing was run against a 
real Snowflake account**. Two checks instead:
   - A stand-in, as in #3208: a small Python HTTP server on `localhost:18099` 
that saves each `POST /session/v1/login-request` (URL and gunzipped JSON) and 
answers with Snowflake's login-failure JSON. `instanceUrl` was 
`localhost:18099/?ssl=off&account=MYACCOUNT`; these are driver connect-string 
parameters, used for the test only. To compare requests, the request IDs and 
the driver's `CLIENT_ENVIRONMENT` block (timings, JVM details) were left out.
   - A login against Snowflake's real front end, for an account that does not 
exist (`w6dummyorg-noaccount.snowflakecomputing.com`, with the driver's 
`tracing=ALL`). The driver's log shows the request it sent before the 404.
   
   | Case | Kamelet | Result |
   |---|---|---|
   | Options unset, `main` and this branch | sink, source | Login requests 
identical: `login-request?databaseName=SHOP`, `PASSWORD` sent. `camel get 
endpoint --json` shows the same URIs on both: 
`local-sql-snowflake-sink-1://INSERT...?dataSource=%23bean%3AdsBean-2` and 
`local-sql-snowflake-source-1://SELECT...?dataSource=%23bean%3AdsBean-2&delay=2000`
 |
   | `schema: SALES`, `warehouse: COMPUTE_WH`, `role: LOADER` / `READER` | 
sink, source | 
`login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER`
 (sink) and `...&roleName=READER` (source), on every login. The login JSON is 
otherwise unchanged, and the endpoint URI is the same as when they are unset |
   | The three options with `privateKeyFile`, no `databaseName` | sink | 
`login-request?schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER`, 
`AUTHENTICATOR=SNOWFLAKE_JWT`, no `PASSWORD`: the options and the key share the 
bean's `properties` map without clashing |
   | The three options, real Snowflake front end, account that does not exist | 
sink | Driver log: `POST 
https://w6dummyorg-noaccount.snowflakecomputing.com:443/session/v1/login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER&requestId=...`,
 answered `HTTP Response code: 404`. The message fails with 
`CannotGetJdbcConnectionException`. The password is shown as `xxxxxx` in the 
Camel startup log, and nowhere else |
   | Connections per message | sink | 2 messages, 2 login requests |
   | Connections per poll | source | `delay: 2000`, about 8 s: 4 polls, 4 login 
requests |
   
   **MongoDB.** Tested against `mongo:8` as a one-member replica set 
(`--replSet rs0`), with collection `shop.orders`. In each case: start the route 
and insert `A`; stop the route and insert `B`; start the route again and insert 
`C`.
   
   | Case | Run 1 sent | Run 2 sent |
   |---|---|---|
   | `mongodb-changes-stream-source` (this branch; the template is unchanged 
from `main`) | `A` | `C`: `B` is lost |
   | Collection does not exist yet | Fails to start: `Collection orders does 
not exist on the database and createCollection is false` | |
   | Plain `mongodb:` route, 
`consumerType=changeStreams&persistentTailTracking=true&persistentId=w6snow` | 
`A` | `C`: `B` is lost, the option does nothing |
   | Plain route, 
`changeStreamTokenRepository=#class:...FileStateRepository#fileStateRepository('.../tokens.dat')`
 | `A` | `B`, `C`: resumed. The file holds `cs-filerepo/orders={"_data": 
"826AFE10..."}` |
   | Plain route, `changeStreamToken` set to the token of an earlier change | 
`A` | `A` again, `B`, `C`: it starts after the same token each time |
   
   ### Build
   
   The library copies were refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. Then `./mvnw clean verify 
-DskipTests` and `./mvnw clean verify` passed from the root. The build 
regenerated `kamelets-index.json`. `KameletsCatalogTest` needed no change.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to