davsclaus opened a new pull request, #3213:
URL: https://github.com/apache/camel-kamelets/pull/3213
Part of #3170: the Azure Storage Blob Kamelets can log in with a SAS token.
`azure-storage-blob-source` can read only some of the blobs, by prefix or
regular expression, and move each blob to another container once it has been
read. `azure-storage-blob-event-based-source` can read the events from a
Service Bus topic, and process several at the same time.
### The options
| Kamelet | Option | Template mapping | What it does |
|---|---|---|---|
| source, sink, append sink | `sasToken` | `sasToken: "RAW({{?sasToken}})"`
(source, and its `deleteBlob` step); `&sasToken=RAW({{?sasToken}})` (sinks) | A
SAS token for the account or the container, used with the new `credentialType`
value `AZURE_SAS`. |
| source | `prefix` | `prefix: "RAW({{?prefix}})"` | Reads only the blobs
whose name starts with it, such as `in/`. |
| source | `regex` | `regex: "RAW({{?regex}})"` | Reads only the blobs whose
whole name matches a Java regular expression. `prefix` is ignored when it is
set. |
| source | `moveAfterRead` | `moveAfterRead: "{{?moveAfterRead}}"` | Moves
each processed blob to `destinationContainer`: it is copied under the same
name, then deleted. |
| source | `destinationContainer` | `destinationContainer:
"{{?destinationContainer}}"` | The container to move to. It is in the same
account and must exist. |
| event-based source | `serviceBusType` | `serviceBusType:
"{{?serviceBusType}}"` | `queue` or `topic`. With `topic`, the events are read
from `subscriptionName`, which was already an option but was not used. |
| event-based source | `maxConcurrentCalls` | `maxConcurrentCalls:
"{{?maxConcurrentCalls}}"` | How many events are processed at the same time. |
- **Defaults:** every new option is optional and has no `default`, as in
#3174. When they are unset, the endpoint URIs are the same as on `main`.
- **`sasToken`:**
- It is `format: password` and in the credentials group.
- Its `pattern` accepts a token with or without the leading `?`. It
rejects a whole SAS URL and an access key pasted by mistake.
- `credentialType` gains `AZURE_SAS` in its enum and its description.
- Each Kamelet's description says which permissions the token needs. The
tests below checked them.
- **`moveAfterRead` together with `deleteAfterRead`:** the template deletes
with a step of its own after the message has been processed. When both options
were true, that step deleted the blob first, so the move failed and the blob
was lost (case M4 below). The step's `precondition` now skips it when
`moveAfterRead` is true, so the blob is moved. With `moveAfterRead` unset, the
expression is true exactly when it was before.
- **`destinationContainer`** has a `pattern` for Azure container names: 3 to
63 lowercase letters, digits and hyphens.
- **`prefix` and `regex`** have no `pattern`. A blob name may hold almost
any character, so a pattern would reject valid values. `RAW()` keeps characters
such as `+` as they are written (case P3).
- **`maxConcurrentCalls`** has `minimum: 1`.
- **Event-based source descriptions:** the summary says it reads from a
queue or a topic, `topicOrQueueName` is now titled "Queue or Topic Name", and
`subscriptionName` says when it is used.
- **Partial pages:**
- All three blob pages list `AZURE_SAS` and have a "Log in with a SAS
Token" section.
- The source page has an example with `prefix`, `moveAfterRead` and a SAS
token.
- The event-based source page has an example that reads from a topic.
#### Left out
- **`maxResultsPerPage` (source):** it sets only the page size of the list
request. `BlobContainerClientWrapper.listBlobs` reads every page, so a poll
still reads every blob. With `maxResultsPerPage=1` and 3 blobs, one poll
returned a batch of 3. To limit how many blobs a poll reads, Camel needs a
change first, such as having `BlobConsumer` honour `maxMessagesPerPoll`.
- **`blobType` and `createPageBlob` (sinks):**
- The sink always uploads a block blob (`operation=uploadBlockBlob`), and
the append sink always appends (`operation=commitAppendBlob`). `blobType` does
not change either operation. In the producer, it only changes the range a
download reads.
- `createPageBlob` is used only by `uploadPageBlob`, which needs a
`PageRange` object in a header, and a message cannot carry one without code.
- Page blobs, which hold disk images in 512-byte pages, do not fit a sink
fed by messages.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT.
- **Blob Storage:** Azurite 3.37.0
(`mcr.microsoft.com/azure-storage/azurite`) served HTTPS with a self-signed
certificate on port 443.
- The JVM ran with `-Djdk.net.hosts.file`, which maps
`devstoreaccount1.blob.core.windows.net` to it, and a trust store. Azurite
takes the account from the host name.
- So the Kamelets ran through the component's real client path:
`https://<accountName>.blob.core.windows.net` with the configured credential.
No `serviceClient` bean was involved.
- The SAS tokens were account and container SAS tokens, computed with the
Azurite account key. Azurite checks their signature and permissions.
- **Service Bus:** Floci AZ 0.16.0 (`floci/floci-az`, with its Artemis-based
AMQP data plane) held a queue, and a topic with a subscription. A route in the
same integration sent Event Grid `BlobCreated` events to it.
| # | Case | Result |
|---|---|---|
| U1 | All options unset, source, sink and append sink, `main` and this
branch | Raw endpoint URIs identical:
`...?accessKey=RAW(...)&credentialType=SHARED_ACCOUNT_KEY&delay=2000`,
`...&operation=uploadBlockBlob`,
`...&blobType=appendBlob&...&operation=commitAppendBlob` |
| U2 | Source with `deleteAfterRead: true`, `main` and this branch |
Consumer and `deleteBlob` endpoint URIs identical. The blobs are deleted on
both |
| U3 | Event-based source unset (queue), `main` and this branch | Endpoint
URIs identical:
`azure-servicebus://blobqueue?connectionString=RAW(...)&serviceBusReceiveMode=RECEIVE_AND_DELETE`.
Events received |
| S1 | `AZURE_SAS`, account token `sp=rwdlacup`, all three Kamelets | Sink
writes `sas.txt`, append sink adds 2 lines to `sas.log` (an `AppendBlob`),
source reads `a.txt`. Endpoints show
`credentialType=AZURE_SAS&...&sasToken=xxxxxx` |
| S2 | The same token with a leading `?` | Same as S1 |
| S3 | Read-only token `sp=rl` | Source reads. Sink and append sink fail
with `403 AuthorizationPermissionMismatch` |
| S4 | Token with a wrong signature | All three fail with `403`. The source
logs `Failed polling endpoint` |
| S5 | Sink with `sp=w`, `c` or `cw` | Sink writes. Append sink fails |
| S6 | Append sink with `sp=rw` or `rca` | Works. With `ca`, `c` or `w` it
fails: it needs Read to check whether the blob exists |
| S7 | Container SAS (`sr=c`, `sp=rl`) for the container | Source reads. A
container SAS for another container reads nothing |
| S8 | Source, `sp=rwdl`, `deleteAfterRead: true` | Blobs read and deleted.
The `deleteBlob` endpoint carries `sasToken` |
| S9 | Source, `sp=rl`, `deleteAfterRead: true` | Delete fails with 403, and
the blob is kept and read again: the token reaches the delete step |
| P1 | `prefix: in/` | Reads `in/a.csv`, `in/ab12.csv`, `in/b.txt`, but not
`other/c.csv` or `top.csv` |
| P2 | `regex: .*\.csv` | Reads the four `.csv` blobs, in any folder |
| P3 | `regex: in/[a-z]+[0-9]+\.csv` | Reads only `in/ab12.csv`: the `+` is
kept |
| P4 | `regex: csv` | Reads nothing: the whole name must match |
| P5 | `regex: .*\.txt` with `prefix: other/` | Reads `in/b.txt`: `prefix`
is ignored |
| M1 | `moveAfterRead: true`, `destinationContainer: tmovedone`, `prefix:
in/` | Each blob read once, then in `tmovedone` under the same name
(`in/a.csv`, ...) with the same content, and gone from the source |
| M2 | `moveAfterRead: true` without `destinationContainer` | The route does
not start: `destinationContainer must be set when moveAfterRead is enabled` |
| M3 | `destinationContainer` that does not exist | Copy fails with 404 and
a warning is logged. The blob is kept and read again on each poll |
| M4 | `moveAfterRead` and `deleteAfterRead` both true | Branch: moved, and
no `deleteBlob` endpoint is created. Same Kamelet without the new
`precondition`: the blobs are deleted and the destination stays empty |
| M5 | Move with an `AZURE_SAS` token `sp=rwdl` | Moved |
| M6 | Move with `sp=rld` | Copy fails and the blob is kept. With `sp=rlwd`
it is moved |
| E1 | `serviceBusType: topic`, `subscriptionName: kamelet` | 4 events sent
to the topic are received. Endpoint
`azure-servicebus://blobtopic?...&serviceBusType=topic&subscriptionName=kamelet`
|
| E2 | Topic and subscription without `serviceBusType`, which is all `main`
can do | Nothing is received: it reads a queue named `blobtopic` |
| E3 | `serviceBusType: topic` without `subscriptionName` | The route does
not start: `topicName (blobtopic) must have a subscriptionName associated with
it` |
| E4 | Topic, `maxConcurrentCalls` unset, a step that blocks for 2 s | One
event at a time: each starts when the previous one ends |
| E5 | Topic, `maxConcurrentCalls: 4`, 12 events | Four events start at the
same millisecond and end 2 s later, in three rounds. Endpoint has
`maxConcurrentCalls=4` |
| E6 | Topic and `getBlob: true`, a `BlobCreated` event for `evt/new.txt` |
The body is the content of the blob, `content of the new blob`, read from
Azurite |
The event-based source was tested end to end against Floci AZ, an emulator.
It was not tested against Azure Service Bus itself.
### Found on the way, not changed here
**`deleteAfterRead: true` on `azure-storage-blob-source` fails every message
after deleting its blob.**
- It happens on `main` too (case U2): the blob is deleted, and then each
exchange logs `No type converter available to convert from type:
java.lang.Boolean to the required type: byte[]` and the warning "Will attempt
to process the message on next poll".
- The cause:
- The `deleteBlob` step leaves `true` as the body.
- The Kamelet's output data type, `binary`, then converts the body at the
end of the route.
- A fix could use the consumer's own `deleteAfterRead`, as `moveAfterRead`
does. Since Camel 4.24 the consumer also takes the blob name before routing, so
a sink that changes `CamelAzureStorageBlobBlobName` no longer makes the delete
miss (noted in #3150). That would add `deleteAfterRead=false` to the URI when
the option is unset, so it belongs in a change of its own.
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the four Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`. `KameletsCatalogTest` needed no count changes.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]