davsclaus opened a new pull request, #3234:
URL: https://github.com/apache/camel-kamelets/pull/3234

   Part of #3170: the SQS and SNS Kamelets get the options the issue lists. 
`aws-sqs-source` can process messages in parallel and keep a slow message 
hidden while it is processed. The SQS sinks can send without listing the 
queues, delay messages, and choose what happens when a message has too many 
headers. The SNS sinks can set a subject and a text for each kind of subscriber.
   
   ### The options
   
   `aws-sqs-source`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `concurrentConsumers` | `concurrentConsumers: "{{?concurrentConsumers}}"` 
| How many threads poll the queue and process messages at the same time |
   | `extendMessageVisibility` | `extendMessageVisibility: 
"{{?extendMessageVisibility}}"` | Keeps a message hidden for as long as it is 
processed. It needs `visibilityTimeout`: every half of it, the timeout is 
extended by `visibilityTimeout` seconds. Without `visibilityTimeout`, the 
integration does not start |
   | `queueOwnerAWSAccountId` | `queueOwnerAWSAccountId: 
"{{?queueOwnerAWSAccountId}}"` | The queue URL is built from the account, the 
region and the name, without ListQueues. Use it for a queue of another account 
too |
   
   `aws-sqs-sink`, `aws-sqs-batch-sink`, `aws-sqs-fifo-sink`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `queueURL` | `queueUrl: "{{?queueURL}}"` | Sends to this URL, without 
ListQueues. Same name as the existing option of `aws-sqs-source` |
   | `queueOwnerAWSAccountId` | `queueOwnerAWSAccountId: 
"{{?queueOwnerAWSAccountId}}"` | As in the source |
   | `delaySeconds` (not in the FIFO sink) | `delaySeconds: 
"{{?delaySeconds}}"` | Keeps each message (each part, in the batch sink) hidden 
for 0 to 900 seconds after it is sent |
   | `messageHeaderExceededLimit` | `messageHeaderExceededLimit: 
"{{?messageHeaderExceededLimit}}"` | `WARN` (the component default), 
`WARN_ONCE`, `IGNORE` or `FAIL`, for a message with more than 10 headers to 
send as message attributes |
   | `serverSideEncryptionEnabled` | `serverSideEncryptionEnabled: 
"{{?serverSideEncryptionEnabled}}"` | With `kmsMasterKeyId`, sets the queue to 
encrypt its messages with that KMS key |
   | `kmsMasterKeyId` | `kmsMasterKeyId: "{{?kmsMasterKeyId}}"` | The KMS key 
ID, alias or ARN |
   
   `aws-sns-sink`, `aws-sns-fifo-sink`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `subject` | `subject: "RAW({{?subject}})"` | The subject of every 
notification. The `subject` / `ce-subject` header still takes precedence |
   | `messageStructure` | `messageStructure: "{{?messageStructure}}"` | `json`: 
the body is a JSON object with a `default` text and a text for each protocol |
   | `serverSideEncryptionEnabled` | `serverSideEncryptionEnabled: 
"{{?serverSideEncryptionEnabled}}"` | With `kmsMasterKeyId`, a topic that 
`autoCreateTopic` creates is encrypted with that KMS key |
   | `kmsMasterKeyId` | `kmsMasterKeyId: "{{?kmsMasterKeyId}}"` | The KMS key 
ID, alias or ARN |
   
   - **Defaults:** all the options are optional and have no `default`, as in 
#3174. When they are unset, the optional placeholders drop them, and the 
endpoint URIs are the same as on `main`. In the FIFO sinks, the options are 
added to both `aws2-sqs` / `aws2-sns` steps.
   - **How the queue is found today:** without a queue URL, `aws2-sqs` calls 
ListQueues with the queue name as prefix when the endpoint starts, so the 
credentials need `sqs:ListQueues`. It does not call it when `queueUrl` is set. 
It also does not call it when the account is known, from 
`queueOwnerAWSAccountId` or from an ARN in `queueNameOrArn`: it then builds 
`https://sqs.<region>.amazonaws.com/<account>/<name>`. The ARN case already 
worked on `main`, and the descriptions now say so.
   - **`serverSideEncryptionEnabled` and `kmsMasterKeyId`, SQS:** they apply to 
an existing queue too, not only to `autoCreateQueue`. When the endpoint starts, 
`aws2-sqs` sends `SetQueueAttributes` with `KmsMasterKeyId` to an existing 
queue (this needs `sqs:SetQueueAttributes`), or `CreateQueue` with it when 
`autoCreateQueue` creates the queue. They change the queue itself, and the 
descriptions say so. `serverSideEncryptionEnabled` without a key does nothing; 
`kmsMasterKeyId` without `serverSideEncryptionEnabled` does nothing either.
   - **`serverSideEncryptionEnabled` and `kmsMasterKeyId`, SNS:** they apply 
only to `CreateTopic`, when `autoCreateTopic` is true and the topic does not 
exist. An existing topic is not changed. `serverSideEncryptionEnabled` without 
a key fails the start when the topic must be created.
   - **The subject:** the Kamelets already took the subject from the `subject` 
or `ce-subject` header and set `CamelAwsSnsSubject` from it. The new `subject` 
option maps to the `subject` endpoint option, which `aws2-sns` uses only when 
that header is missing, so the header still wins. It is in `RAW()` so that `+`, 
`%`, `&` and `#` are kept. The `pattern:` `^[ -~]+$` (printable ASCII, no line 
breaks) follows the rules SNS has for a subject. `aws2-sns` cuts a subject 
longer than 100 characters to 100.
   - **Why the FIFO SQS sink has no `delaySeconds`:** SQS takes no delay for 
each message on a FIFO queue, only a delay for the whole queue. `aws2-sqs` 
skips the per-message delay for a queue whose name ends in `.fifo`. The 
description says so.
   - **`messageHeaderExceededLimit`:** the 10 headers that are sent are the 
first 10 in the order they were set on the message, not in alphabetical order. 
In the batch sink the headers go with each part, so `WARN_ONCE` logs once for 
each part.
   - **`pattern:`** on the new free-form options: `queueURL` 
(`^https?://[^\s]+$`), `queueOwnerAWSAccountId` (12 digits), `kmsMasterKeyId` 
(as `awsKMSKeyId` in the S3 Kamelets), `subject`.
   - **Descriptions:** the "Good to know" paragraphs, the option descriptions 
and the `subject` header descriptions are updated. Each partial page has new 
sections with an example: "Slow Messages and Parallel Processing" and "Without 
Permission to List the Queues" (source); "Delaying the Messages", "Without 
Permission to List the Queues", "Too Many Headers" and "Encrypting the Queue" 
(SQS sinks); "The Subject", "A Text for Each Kind of Subscriber" and 
"Encrypting a New Topic" (SNS sinks).
   - `KameletsCatalogTest` counts the options of `aws-sqs-source`: now 29.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci`. To see the 
AWS calls, the Kamelets connected to a small reverse proxy in front of floci. 
It logged the operation and the body of each request. In the ListQueues cases, 
it answered ListQueues with `AccessDenied`.
   
   | Case | Result |
   |---|---|
   | Options unset, all six Kamelets, `main` and this branch | `camel get 
endpoint`: the 8 `aws2-sqs` / `aws2-sns` URIs are identical; the messages are 
sent and received |
   | ListQueues denied, `aws-sqs-sink` without `queueURL` | Fails to start: 
`ListQueues` is called and denied (`not authorized to perform ListQueues`) |
   | ListQueues denied, `queueURL` on `aws-sqs-sink`, `aws-sqs-batch-sink`, 
`aws-sqs-fifo-sink` | All three send; the proxy logs only `SendMessage` / 
`SendMessageBatch` to that URL, with no `ListQueues` |
   | ListQueues denied, `queueOwnerAWSAccountId: "000000000000"` on 
`aws-sqs-sink` and `aws-sqs-source` | The sink sends and the source receives 
and deletes, with no `ListQueues`. Queue URL 
`https://sqs.us-east-1.amazonaws.com/000000000000/qb` |
   | ListQueues denied, ARN in `queueNameOrArn`, `aws-sqs-sink` (no new option) 
| Sends, with no `ListQueues` |
   | `delaySeconds: 10` on `aws-sqs-sink` | Received 10.27 s after 
`SentTimestamp` |
   | `delaySeconds: 6` on `aws-sqs-batch-sink`, 2 parts | Both parts received 
6.17 s after `SentTimestamp` |
   | No `delaySeconds`, same queue | Received 12 ms after `SentTimestamp` |
   | `aws-sqs-source`, 6 messages, each takes 2 s (blocking delay), 
`concurrentConsumers` unset | One thread, one message after the other: 14.6 s 
in all |
   | The same with `concurrentConsumers: 3` | Three threads, three messages at 
a time: 4.5 s in all |
   | `visibilityTimeout: 4`, a message that takes 12 s, 2 threads, no 
`extendMessageVisibility` | Received twice: by the second thread 4.1 s later, 
with `ApproximateReceiveCount` 2, and processed twice |
   | The same with `extendMessageVisibility: true` | Received once; 
`ChangeMessageVisibilityBatch` every 2 s until it ended, then `DeleteMessage` |
   | `extendMessageVisibility: true` without `visibilityTimeout` | Does not 
start: `extendMessageVisibility requires visibilityTimeout to be set on the 
Endpoint` |
   | 12 headers, `messageHeaderExceededLimit` unset, `aws-sqs-sink` | Sent with 
10 attributes (`zeta`, `Beta`, `alpha`, `h04`…`h10`, the first 10 set); a 
warning each for `h11` and `h12` |
   | The same, `WARN_ONCE` / `IGNORE` | The same 10 attributes; one warning / 
no warning |
   | The same, `WARN_ONCE`, `aws-sqs-batch-sink`, 2 parts | One warning for 
each part |
   | The same, `FAIL`, `aws-sqs-sink` and `aws-sqs-fifo-sink` | 
`IllegalArgumentException: Number of message headers exceeded`; no 
`SendMessage` |
   | `serverSideEncryptionEnabled: true`, `kmsMasterKeyId: alias/aws/sqs`, 
existing queue, `aws-sqs-sink` | `SetQueueAttributes 
{"KmsMasterKeyId":"alias/aws/sqs"}` at start; the queue has the key |
   | `kmsMasterKeyId` only, existing queue | No `SetQueueAttributes`; the queue 
has no key |
   | Both, `autoCreateQueue: true`, new queue, `aws-sqs-batch-sink` and 
`aws-sqs-fifo-sink` | `CreateQueue` with `KmsMasterKeyId` (and `FifoQueue` for 
the FIFO one); the queues have the key |
   | `aws-sns-sink`, `subject: "Order #42 + 50% off & more"`, SQS queue 
subscribed | The notification in the queue has `"Subject":"Order #42 + 50% off 
& more"` |
   | `subject` option and `subject` header `From the header` | `"Subject":"From 
the header"` |
   | Neither | No `Subject` in the notification |
   | `messageStructure: json`, body `{"default": …, "sqs": "case4 text for 
sqs", "email": …}` | `Publish … MessageStructure=json`; the queue gets 
`"Message":"case4 text for sqs"` |
   | `messageStructure: json`, body without `default` | 
`InvalidParameterException: Message Structure - No default entry in JSON 
message body` |
   | `aws-sns-fifo-sink`, `subject`, FIFO queue subscribed | `"Subject":"Fifo 
subject"` |
   | `aws-sns-fifo-sink`, `messageStructure: json` | The queue gets the `sqs` 
text |
   | `serverSideEncryptionEnabled` + `kmsMasterKeyId`, `autoCreateTopic`, new 
topic, `aws-sns-sink` / `aws-sns-fifo-sink` | `CreateTopic` with 
`KmsMasterKeyId`; the topics have the key |
   | The same, existing topic | No `CreateTopic` or `SetTopicAttributes`; the 
topic has no key |
   | `serverSideEncryptionEnabled` without `kmsMasterKeyId`, new topic | Does 
not start: `The option kmsMasterKeyId is required when 
serverSideEncryptionEnabled is true` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the six Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to