davsclaus opened a new pull request, #3217:
URL: https://github.com/apache/camel-kamelets/pull/3217

   Part of #3170: the Azure Storage Queue, Data Lake and Files Kamelets take 
the component options that people need most and that the Kamelets did not 
offer: creating a missing queue and the visibility timeout for the queues; 
choosing a directory and the files to read, and logging in with an account key 
or a SAS token, for the Data Lake; filters, moving the files read and 
remembering them across restarts, and what to do when a file exists, for Azure 
Files.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | `azure-storage-queue-sink` | `createQueue` | 
`&createQueue={{?createQueue}}` | Creates the queue if it does not exist, 
before each message. Without it, a missing queue fails the message. |
   | `azure-storage-queue-sink` | `visibilityTimeout` | 
`&visibilityTimeout={{?visibilityTimeout}}` | Hides each message for that long 
after it is added, such as `PT10M`, so that readers receive it later. |
   | `azure-storage-queue-source` | `visibilityTimeout` | 
`&visibilityTimeout={{?visibilityTimeout}}` | How long a received message stays 
hidden, so how soon a failed message is received again. 30 seconds when not 
set. |
   | `azure-storage-queue-source` | `delay` | `&delay={{?delay}}` | 
Milliseconds between two checks of the queue. 500 when not set. `delay` is not 
a name the `kamelet:` component takes (the CatalogValidator list), and 
`azure-storage-datalake-source` and `azure-storage-files-source` already have 
it. |
   | `azure-storage-datalake-source` | `directoryName` | `path: 
"RAW({{?directoryName}})"` | The directory to read from. The component's option 
is `path`; the name `directoryName` matches the sink and the Files Kamelets. |
   | `azure-storage-datalake-source` | `recursive` | `recursive: 
"{{?recursive}}"` | Also reads the directories below. Before, only the files at 
the top of the file system were read. |
   | `azure-storage-datalake-source` | `regex` | `regex: "RAW({{?regex}})"` | 
Reads only the files whose path, from the top of the file system, matches as a 
whole (case-sensitive). |
   | `azure-storage-datalake-source` | `fileName` | `fileName: 
"RAW({{?fileName}})"` | Reads only this one file; `directoryName`, `recursive` 
and `regex` are not used then. |
   | `azure-storage-datalake-sink` | `directoryName` | a `choice` step: when 
`'{{?directoryName}}' != ''`, `CamelAzureStorageDataLakeFileName` becomes 
`{{?directoryName}}/${header[CamelAzureStorageDataLakeFileName]}` | Writes the 
files into a directory. |
   | both Data Lake Kamelets | `accountKey` | `accountKey: 
"RAW({{?accountKey}})"` | Logs in with an account key, with `credentialType: 
SHARED_KEY_CREDENTIAL`. |
   | both Data Lake Kamelets | `sasSignature` | `sasSignature: 
"RAW({{?sasSignature}})"` | Logs in with a SAS token, with `credentialType: 
AZURE_SAS`. |
   | `azure-storage-files-source` | `maxMessagesPerPoll` | 
`&maxMessagesPerPoll={{?maxMessagesPerPoll}}` | The most files to read at each 
poll. |
   | `azure-storage-files-source` | `include` | `&include=RAW({{?include}})` | 
A regular expression on the file name (case-insensitive). |
   | `azure-storage-files-source` | `antInclude` | 
`&antInclude=RAW({{?antInclude}})` | Wildcards on the path below 
`directoryName`, such as `**/*.csv`. |
   | `azure-storage-files-source` | `move` | `&move=RAW({{?move}})` | Where to 
move each file once read, such as `.done`. |
   | `azure-storage-files-source` | `noop` | `&noop={{?noop}}` | Leaves the 
files in place and reads each file once (the list is kept in memory). |
   | `azure-storage-files-source` | `idempotentRepository` | 
`&idempotentRepository=#bean:{{?idempotentRepository}}` | The name of a bean, 
such as a `FileIdempotentRepository`, that keeps the list of the files read, so 
that it survives a restart. Used with `noop`. |
   | `azure-storage-files-sink` | `fileExist` | `&fileExist={{?fileExist}}` | 
`Override` (the component default), `Fail` or `Ignore`. |
   
   - **Defaults:** all options are optional and have no `default`, as in #3174. 
When they are unset, the optional placeholders drop them, and the endpoint URIs 
and the configured values are the same as on `main` (see the first rows below).
   - **`clientId`, `clientSecret` and `tenantId` are no longer required** in 
the two Data Lake Kamelets, so that `accountKey`, `sasSignature` or 
`AZURE_IDENTITY` work without dummy values. They are mapped as `{{?clientId}}`, 
`RAW({{?clientSecret}})` and `{{?tenantId}}`. With the default `CLIENT_SECRET` 
and without them, the integration fails to start with `Must provide non-null 
values for clientId, tenantId, clientSecret properties in 
ClientSecretCredentialBuilder`. Their descriptions now say "Required when 
`credentialType` is `CLIENT_SECRET`, the default".
   - **Why the sink's `directoryName` is not the component's `directoryName`:** 
the `upload` operation of the Data Lake producer does not use it (only 
`createFile` and `deleteDirectory` do), so the Kamelet puts the directory in 
front of the file name. A `file` header with a path, such as `2026/q1.csv`, 
ends up below the directory. `pattern` keeps it free of a leading or trailing 
`/`, `'` and `$`, as the value goes into a simple expression.
   - **`idempotentRepository`** is mapped as `#bean:...` as in #3178; the bean 
is declared in the integration (see the partial page). Unlike FTP, 
`azure-files` has `idempotent=false` by default, so the repository is used only 
with `noop: true`; without it, Camel logs that it is not used, and the 
description says so.
   - **`RAW()`:** used for the free-form strings (`regex`, `include`, 
`antInclude`, `move`, `directoryName`, `fileName`) and the secrets. Values with 
`+`, `%2B`, `&`, spaces and `${...}` reached the endpoint unchanged (see below).
   - **`pattern`:** the two `visibilityTimeout` options take ISO-8601 durations 
(`^P(\d+D)?(T(\d+H)?(\d+M)?(\d+(\.\d+)?S)?)?$`). At runtime Camel also takes 
`5s`, and reads a plain number as milliseconds, which the descriptions say. The 
Data Lake `directoryName` and `fileName` have patterns too.
   - **Left out:** deleting the files after reading in 
`azure-storage-datalake-source`. The Data Lake consumer has no option for it; 
it needs a change in Camel first. `fileExist` offers only `Override`, `Fail` 
and `Ignore`: the component rejects `Append` and `Move`, and `TryRename` needs 
`tempFileName`.
   - **Descriptions:** the "Good to know" paragraphs point to the new options. 
The queue source now says that the 30 seconds before a failed message comes 
back is the visibility timeout, and that a message still in processing after 
that time can be received again.
   - **Partial pages:** an example for each Kamelet, and a new page for 
`azure-storage-datalake-source`.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT.
   - **Queue:** end to end against Azurite 
(`mcr.microsoft.com/azure-storage/azurite`), with a `QueueServiceClient` bean 
on the Azurite connection string, which the component autowires.
   - **Data Lake:** Azurite has no Data Lake (dfs) API, and I found no local 
service that has one. The component always builds 
`https://<account>.dfs.core.windows.net`, but it autowires a 
`DataLakeServiceClient` bean. So I tested against a small recording stand-in of 
the dfs API, about 150 lines of Python: it keeps the files in memory, answers 
List Paths, Create, Append and Flush and the reads, and logs every request with 
its query. It shows what the Kamelet sends. It models the directory listing of 
the service, and the real listing may differ in details. With the bean, the 
component uses the bean's client and its login, so `accountKey` and 
`sasSignature` were checked by the configured values, and by whether the 
component can build a client from them.
   - **Files:** there is no local Azure Files service (Azurite has none), and 
the component always uses `https://`. The Files options were checked by the 
values the endpoint is configured with, read from the endpoint objects by a 
small Java route, and by what the component does at start-up. They were not run 
against a share.
   
   | Case | Result |
   |---|---|
   | All new options unset, all six Kamelets, `main` and this branch | `camel 
get endpoint --json` lists the same URIs, and the configured endpoint values 
are the same |
   | All options set, with `+`, `%2B`, `&`, a space and `${...}` in the values 
| The endpoints are configured with the values as written, such as 
`sasSignature=sv=2022-11-02&ss=b&sig=ab+c%2Bd`, `path=exports/2026 q1`, 
`move=done/${date:now:yyyyMMdd}/${file:name}`, and a `FileIdempotentRepository` 
for `idempotentRepository` |
   | Queue sink, missing queue, `createQueue` unset | Fails: `QueueNotFound` |
   | Queue sink, missing queue, `createQueue: true` | Queue created and message 
sent; a queue source then received it. A second run, with the queue there, also 
sent |
   | Queue sink, `visibilityTimeout` unset | Received 0.2 s after sending |
   | Queue sink, `visibilityTimeout: PT10S` | Received 10.4 s after sending |
   | Queue sink, `visibilityTimeout: 5s` | Received 5.3 s after sending |
   | Queue sink, `visibilityTimeout: 600` | Received at once (600 ms) |
   | Queue sink, `visibilityTimeout: P8D` | Fails: status 400 |
   | Queue sink, `visibilityTimeout: PT20S` with `expiration: PT10S` | Fails: 
`InvalidQueryParameterValue` |
   | Queue source, a step that always fails, `visibilityTimeout` unset | 
Received again after 30 s, dequeue count 2 |
   | Queue source, the same with `visibilityTimeout: PT5S` | Received again 
every 5 s, dequeue count 1, 2, 3, 4 |
   | Queue source, `visibilityTimeout: PT3S`, a step that waits 8 s without 
holding up the source (`delay` EIP) | Received again every 3 s while the first 
was still processed; the deletes then fail with `PopReceiptMismatch` (as the 
description now says) |
   | The same with a step that waits 8 s and holds up the source | Received 
once, deleted |
   | Queue source, three messages, `delay` unset | Received 0.5 s apart |
   | Queue source, three messages, `delay: 5000` | Received 5 s apart |
   | Data Lake source, unset, `main` and this branch | List Paths with 
`directory=/`, `recursive=false`; read `top.csv`, `top.txt` only |
   | `directoryName: exports` | `directory=exports`; read `exports/a.csv`, 
`exports/b.txt`, not `exports/2026/c.csv`; `CamelAzureStorageDataLakeFileName` 
is the full path |
   | `directoryName: exports`, `recursive: true` | `recursive=true`; also read 
`exports/2026/c.csv` |
   | `recursive: true` | All six files |
   | `recursive: true`, `regex: .*\.csv` | The four CSV files, in all 
directories |
   | `recursive: true`, `regex: exports/.*\.csv` | `exports/a.csv`, 
`exports/2026/c.csv` |
   | `regex: .*\.CSV` | Nothing (case-sensitive) |
   | `fileName: exports/2026/c.csv`, with `directoryName` and `regex` that do 
not match it | No listing; read only `exports/2026/c.csv` |
   | `fileName` of a missing file | Each poll fails with a WARN 
(`BlobStorageException` 404), nothing read |
   | Data Lake sink, unset, `main` and this branch | `sales.csv`, `2026/q1.csv` 
and `<exchangeId>` at the top of the file system |
   | Data Lake sink, `directoryName: exports/2026`, without `clientId`, 
`clientSecret`, `tenantId` | `exports/2026/sales.csv`, 
`exports/2026/2026/q1.csv`, `exports/2026/<exchangeId>` |
   | Data Lake sink, default `CLIENT_SECRET` without `clientId`, 
`clientSecret`, `tenantId` | Fails to start, naming the three options |
   | `SHARED_KEY_CREDENTIAL` with `accountKey`; `AZURE_SAS` with `sasSignature` 
| Starts, so the client was built from the key or the token; the message then 
stays in flight, as the account does not exist |
   | `SHARED_KEY_CREDENTIAL` without `accountKey`; `AZURE_SAS` without 
`sasSignature` | Fails to start: `client cannot be null` |
   | Files source, `move` with `deleteAfterRead: true` | Fails to start: `You 
cannot both set delete=true and move options` |
   | Files source, `idempotentRepository` without `noop` | Starts, with a WARN 
that the repository is not used |
   | Files source, `idempotentRepository` with `noop: true` | Starts, no WARN |
   | Files source, `noop: true` with `move` | Starts, with a WARN that `noop` 
wins and the files are not moved |
   | Files source, `idempotentRepository` naming a missing bean | Fails to 
start |
   | Files sink, `fileExist: Fail` | The endpoint has `fileExist=Fail` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the six Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. `KameletsCatalogTest` needed no change.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to