davsclaus opened a new pull request, #3237:
URL: https://github.com/apache/camel-kamelets/pull/3237
Part of #3170: `snowflake-source` and `snowflake-sink` can choose the
warehouse, the schema and the role of the session.
`mongodb-changes-stream-source` gets no new option: it cannot resume after a
restart without a bean. Its docs now say that.
### The options
`snowflake-source` and `snowflake-sink`. They are set on the data source
bean `dsBean`, next to `databaseName`. The bean is
`SnowflakeDataSourceFactory.createDataSource()`, as before #3208 and after it.
| Option | Template mapping (bean `dsBean`) | What it does |
|---|---|---|
| `warehouse` | `warehouse: "{{?warehouse}}"` | The virtual warehouse that
runs the queries or statements. |
| `schema` | `schema: "{{?schema}}"` | The schema of `databaseName` that
holds the tables. |
| `role` | `role: "{{?role}}"` | The role to run as, which must be granted
to the user. |
- **Bean setters, not `properties[...]`:** snowflake-jdbc 4.3.4 has the
setters `setWarehouse`, `setSchema` and `setRole` on
`SnowflakeBasicDataSource`. Each one puts its `SFSessionProperty` key into the
same `properties` map that #3208 fills with the key options. So these map like
`databaseName` does. #3208 used `properties[...]` only because the key setters
take two arguments.
- **What reaches Snowflake:** the driver sends them as the `warehouse`,
`schemaName` and `roleName` query parameters of `POST
/session/v1/login-request`, next to `databaseName`.
- **Optional, no `default`:** with `{{?x}}` unset, the setter is not called.
The login request and the endpoint URI are then identical to `main`'s; see the
table below. Snowflake then uses the user's `DEFAULT_WAREHOUSE`,
`DEFAULT_NAMESPACE` and `DEFAULT_ROLE`.
- **No `pattern:`:** none of them goes into a URI. The bean gets them as
they are, and the driver encodes them as query parameters.
- **Names:** `warehouse`, `schema` and `role` are the driver's names (the
JDBC connection parameters, and the `SnowflakeDataSource` setters). None of
them is an option name that the `kamelet:` component takes.
- **Descriptions:**
- "Good to know" of both Kamelets names the three options and says that
the user's defaults apply when they are not set.
- Each option says what happens when it is not set.
- Both partial pages have a "Warehouse, Schema and Role" section with an
example, and a "Connections" section (see below).
- The statements about Snowflake's defaults and about a statement failing
with no warehouse come from Snowflake's [JDBC connection
parameters](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-parameters)
and user documentation. They were **not** run against Snowflake.
### Left out
- **Pool settings (`maxTotal`, `maxIdle`, `maxWaitMillis`)**, as #3215 adds
them to the other SQL Kamelets. There, the bean is a commons-dbcp2
`BasicDataSource`. Here it is the driver's `SnowflakeBasicDataSource`, which
does not pool: each `getConnection()` logs in and opens a new Snowflake
session, and closing the connection ends it. In the tests below, the sink
logged in once per message (2 messages, 2 logins) and the source once per poll
(4 polls, 4 logins). The "Good to know" paragraphs and the partial pages now
say so.
- A pool would mean a different bean: a commons-dbcp2 `BasicDataSource`
with the Snowflake driver class, with the key options passed through
`connectionProperties`. Long-lived pooled sessions also raise
Snowflake-specific questions, such as session expiry and keep-alive, and that
needs its own PR and its own tests. `commons-dbcp2` is still in
`spec.dependencies` from before #3055, but nothing uses it.
- **Resuming `mongodb-changes-stream-source` after a restart.**
camel-mongodb 4.24.0-SNAPSHOT (the same in 4.23.0) has three options that look
related. None of them works in this Kamelet without a bean:
- `changeStreamTokenRepository`: this is what resumes. After each
exchange, the consumer saves the change's resume token in a `StateRepository`,
keyed by route ID and collection, and starts after it on the next start. It
takes an object, such as a `FileStateRepository`. A Kamelet can declare a bean,
but it creates the bean every time, and the endpoint must name it every time.
An optional placeholder drops a URI parameter only when the value starts with
the placeholder, so
`#class:...FileStateRepository#fileStateRepository('{{?file}}')` is not dropped
when unset. The option cannot be made optional with the endpoint unchanged from
`main`.
- `changeStreamToken`: a fixed token to start after. It is read on
**every** start, and it wins over the repository, so each restart sends the
same changes again (see the table). The Kamelet does not send the tokens as
text either: the only header that carries one, `CamelOffset`, holds a Java
object. So a user has no token to set.
- `persistentTailTracking` (with `persistentId`): only the tailable cursor
consumer (`mongodb-source`) uses it. With `consumerType=changeStreams`, it does
nothing (see the table).
Instead, the "Good to know" paragraph and a new "Restarts" section of the
partial page say that the changes made while the Kamelet is stopped are never
sent. They also say that the collection must exist when the Kamelet starts;
otherwise it fails with `Collection orders does not exist on the database and
createCollection is false`, which happened in a test. Supporting resume would
need a Camel change first, such as a camel-mongodb option that takes a file
path or that stores the tokens in a MongoDB collection.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT, `camel run ... --local-kamelet-dir`.
**Snowflake.** There is no local Snowflake, and **nothing was run against a
real Snowflake account**. Two checks instead:
- A stand-in, as in #3208: a small Python HTTP server on `localhost:18099`
that saves each `POST /session/v1/login-request` (URL and gunzipped JSON) and
answers with Snowflake's login-failure JSON. `instanceUrl` was
`localhost:18099/?ssl=off&account=MYACCOUNT`; these are driver connect-string
parameters, used for the test only. To compare requests, the request IDs and
the driver's `CLIENT_ENVIRONMENT` block (timings, JVM details) were left out.
- A login against Snowflake's real front end, for an account that does not
exist (`w6dummyorg-noaccount.snowflakecomputing.com`, with the driver's
`tracing=ALL`). The driver's log shows the request it sent before the 404.
| Case | Kamelet | Result |
|---|---|---|
| Options unset, `main` and this branch | sink, source | Login requests
identical: `login-request?databaseName=SHOP`, `PASSWORD` sent. `camel get
endpoint --json` shows the same URIs on both:
`local-sql-snowflake-sink-1://INSERT...?dataSource=%23bean%3AdsBean-2` and
`local-sql-snowflake-source-1://SELECT...?dataSource=%23bean%3AdsBean-2&delay=2000`
|
| `schema: SALES`, `warehouse: COMPUTE_WH`, `role: LOADER` / `READER` |
sink, source |
`login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER`
(sink) and `...&roleName=READER` (source), on every login. The login JSON is
otherwise unchanged, and the endpoint URI is the same as when they are unset |
| The three options with `privateKeyFile`, no `databaseName` | sink |
`login-request?schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER`,
`AUTHENTICATOR=SNOWFLAKE_JWT`, no `PASSWORD`: the options and the key share the
bean's `properties` map without clashing |
| The three options, real Snowflake front end, account that does not exist |
sink | Driver log: `POST
https://w6dummyorg-noaccount.snowflakecomputing.com:443/session/v1/login-request?databaseName=SHOP&schemaName=SALES&warehouse=COMPUTE_WH&roleName=LOADER&requestId=...`,
answered `HTTP Response code: 404`. The message fails with
`CannotGetJdbcConnectionException`. The password is shown as `xxxxxx` in the
Camel startup log, and nowhere else |
| Connections per message | sink | 2 messages, 2 login requests |
| Connections per poll | source | `delay: 2000`, about 8 s: 4 polls, 4 login
requests |
**MongoDB.** Tested against `mongo:8` as a one-member replica set
(`--replSet rs0`), with collection `shop.orders`. In each case: start the route
and insert `A`; stop the route and insert `B`; start the route again and insert
`C`.
| Case | Run 1 sent | Run 2 sent |
|---|---|---|
| `mongodb-changes-stream-source` (this branch; the template is unchanged
from `main`) | `A` | `C`: `B` is lost |
| Collection does not exist yet | Fails to start: `Collection orders does
not exist on the database and createCollection is false` | |
| Plain `mongodb:` route,
`consumerType=changeStreams&persistentTailTracking=true&persistentId=w6snow` |
`A` | `C`: `B` is lost, the option does nothing |
| Plain route,
`changeStreamTokenRepository=#class:...FileStateRepository#fileStateRepository('.../tokens.dat')`
| `A` | `B`, `C`: resumed. The file holds `cs-filerepo/orders={"_data":
"826AFE10..."}` |
| Plain route, `changeStreamToken` set to the token of an earlier change |
`A` | `A` again, `B`, `C`: it starts after the same token each time |
### Build
The library copies were refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. Then `./mvnw clean verify
-DskipTests` and `./mvnw clean verify` passed from the root. The build
regenerated `kamelets-index.json`. `KameletsCatalogTest` needed no change.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]