This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 16bad2294 chore: the SQS Kamelets can process in parallel, delay
messages and skip ListQueues, the SNS sinks can set a subject (#3170) (#3234)
16bad2294 is described below
commit 16bad22943477d85768eea3fbbf7ba47f3bbe471
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:38:09 2026 +0100
chore: the SQS Kamelets can process in parallel, delay messages and skip
ListQueues, the SNS sinks can set a subject (#3170) (#3234)
aws-sqs-source gets concurrentConsumers, extendMessageVisibility (which
needs visibilityTimeout) and queueOwnerAWSAccountId.
aws-sqs-sink, aws-sqs-batch-sink and aws-sqs-fifo-sink get queueURL and
queueOwnerAWSAccountId, so that the queue is not looked up with
ListQueues, messageHeaderExceededLimit, and serverSideEncryptionEnabled
with kmsMasterKeyId, which change the queue itself: an existing queue is
updated with SetQueueAttributes when the integration starts, and a queue
that autoCreateQueue creates is created with the key. aws-sqs-sink and
aws-sqs-batch-sink also get delaySeconds; aws-sqs-fifo-sink does not, as
a FIFO queue takes no delay for each message and aws2-sqs ignores it.
aws-sns-sink and aws-sns-fifo-sink get subject (the subject header still
takes precedence), messageStructure, and serverSideEncryptionEnabled with
kmsMasterKeyId, which apply only to a topic that autoCreateTopic creates.
All the options are optional, without a default: when they are unset the
endpoint URIs are the same as before. Tested with Camel JBang
4.24.0-SNAPSHOT against floci.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../partials/aws-sns-fifo-sink-description.adoc | 35 +++++++++++++++++-
.../ROOT/partials/aws-sns-sink-description.adoc | 35 +++++++++++++++++-
.../partials/aws-sqs-batch-sink-description.adoc | 34 +++++++++++++++++
.../partials/aws-sqs-fifo-sink-description.adoc | 28 ++++++++++++++
.../ROOT/partials/aws-sqs-sink-description.adoc | 34 +++++++++++++++++
.../ROOT/partials/aws-sqs-source-description.adoc | 34 +++++++++++++++++
kamelets/aws-sns-fifo-sink.kamelet.yaml | 35 ++++++++++++++++--
kamelets/aws-sns-sink.kamelet.yaml | 31 ++++++++++++++--
kamelets/aws-sqs-batch-sink.kamelet.yaml | 43 +++++++++++++++++++++-
kamelets/aws-sqs-fifo-sink.kamelet.yaml | 40 +++++++++++++++++++-
kamelets/aws-sqs-sink.kamelet.yaml | 43 +++++++++++++++++++++-
kamelets/aws-sqs-source.kamelet.yaml | 21 ++++++++++-
.../kamelets/catalog/KameletsCatalogTest.java | 5 ++-
.../main/resources/META-INF/kamelets-index.json | 12 +++---
.../kamelets/aws-sns-fifo-sink.kamelet.yaml | 35 ++++++++++++++++--
.../resources/kamelets/aws-sns-sink.kamelet.yaml | 31 ++++++++++++++--
.../kamelets/aws-sqs-batch-sink.kamelet.yaml | 43 +++++++++++++++++++++-
.../kamelets/aws-sqs-fifo-sink.kamelet.yaml | 40 +++++++++++++++++++-
.../resources/kamelets/aws-sqs-sink.kamelet.yaml | 43 +++++++++++++++++++++-
.../resources/kamelets/aws-sqs-source.kamelet.yaml | 21 ++++++++++-
20 files changed, 614 insertions(+), 29 deletions(-)
diff --git a/docs/modules/ROOT/partials/aws-sns-fifo-sink-description.adoc
b/docs/modules/ROOT/partials/aws-sns-fifo-sink-description.adoc
index 4a894fa97..9a0e73d4c 100644
--- a/docs/modules/ROOT/partials/aws-sns-fifo-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sns-fifo-sink-description.adoc
@@ -23,9 +23,42 @@ For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/de
In the Kamelet you can optionally set the following header:
-- `subject` / `ce-subject`: the subject of the message
+- `subject` / `ce-subject`: the subject of the message. It takes precedence
over the `subject` option.
- `group` / `ce-group`: the message group ID. If you do not set it, each
message gets a group of its own.
+=== The Subject
+
+The `subject` option sets the subject of every notification. A `subject` or
`ce-subject` header on a message takes precedence over it. Email subscribers
see the subject as the subject line, and an SQS queue subscribed to the topic
gets it in the `Subject` field of the JSON notification. A subject is ASCII
text without line breaks; a longer one than 100 characters is cut to 100.
+
+=== A Text for Each Kind of Subscriber
+
+With `messageStructure` set to `json`, the body is a JSON object with a
`default` text for all subscribers and, optionally, a text for each protocol,
such as `sqs`, `email` or `sms`. SNS rejects a body without `default`. For
example, the body
+
+[source,json]
+----
+{"default": "Order 42 received", "email": "Your order 42 was received.",
"sqs": "{\"orderId\": 42}"}
+----
+
+sends `{"orderId": 42}` to the SQS queues subscribed to the topic, and `Your
order 42 was received.` to the email addresses.
+
+=== Encrypting a New Topic
+
+`serverSideEncryptionEnabled` and `kmsMasterKeyId` apply only to a topic that
`autoCreateTopic` creates: the topic is then created to encrypt its messages
with that key of AWS KMS. With `serverSideEncryptionEnabled` and no
`kmsMasterKeyId`, the integration does not start when the topic must be
created. They do not change a topic that exists already.
+
+For example:
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:aws-sns-fifo-sink"
+ parameters:
+ topicNameOrArn: "order-events.fifo"
+ region: "eu-west-1"
+ subject: "Order received"
+ messageStructure: "json"
+----
+
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-sns-sink-description.adoc
b/docs/modules/ROOT/partials/aws-sns-sink-description.adoc
index 0ed67168e..936941ed7 100644
--- a/docs/modules/ROOT/partials/aws-sns-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sns-sink-description.adoc
@@ -23,7 +23,40 @@ For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/de
In the Kamelet you can optionally set the following header:
-- `subject` / `ce-subject`: the subject of the message
+- `subject` / `ce-subject`: the subject of the message. It takes precedence
over the `subject` option.
+
+=== The Subject
+
+The `subject` option sets the subject of every notification. A `subject` or
`ce-subject` header on a message takes precedence over it. Email subscribers
see the subject as the subject line, and an SQS queue subscribed to the topic
gets it in the `Subject` field of the JSON notification. A subject is ASCII
text without line breaks; a longer one than 100 characters is cut to 100.
+
+=== A Text for Each Kind of Subscriber
+
+With `messageStructure` set to `json`, the body is a JSON object with a
`default` text for all subscribers and, optionally, a text for each protocol,
such as `sqs`, `email` or `sms`. SNS rejects a body without `default`. For
example, the body
+
+[source,json]
+----
+{"default": "Order 42 received", "email": "Your order 42 was received.",
"sqs": "{\"orderId\": 42}"}
+----
+
+sends `{"orderId": 42}` to the SQS queues subscribed to the topic, and `Your
order 42 was received.` to the email addresses.
+
+=== Encrypting a New Topic
+
+`serverSideEncryptionEnabled` and `kmsMasterKeyId` apply only to a topic that
`autoCreateTopic` creates: the topic is then created to encrypt its messages
with that key of AWS KMS. With `serverSideEncryptionEnabled` and no
`kmsMasterKeyId`, the integration does not start when the topic must be
created. They do not change a topic that exists already.
+
+For example:
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:aws-sns-sink"
+ parameters:
+ topicNameOrArn: "order-events"
+ region: "eu-west-1"
+ subject: "Order received"
+ messageStructure: "json"
+----
+
=== Through an HTTP Proxy
diff --git a/docs/modules/ROOT/partials/aws-sqs-batch-sink-description.adoc
b/docs/modules/ROOT/partials/aws-sqs-batch-sink-description.adoc
index 9f6274197..09798abe7 100644
--- a/docs/modules/ROOT/partials/aws-sqs-batch-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sqs-batch-sink-description.adoc
@@ -19,6 +19,40 @@ Only one of access key/secret key or default credentials
provider could be used
For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html[AWS
credentials documentation]
+=== Delaying the Messages
+
+`delaySeconds` keeps each message hidden for that many seconds after it is
sent, up to 900 (15 minutes), before a reader can receive it. Use it to process
a message later, such as a retry in a minute.
+
+=== Without Permission to List the Queues
+
+By default the Kamelet finds the address of the queue by listing the queues,
which needs the `sqs:ListQueues` permission. It does not list the queues when
it is given the address:
+
+- **Queue URL** (`queueURL`): the full address of the queue, as the SQS
console shows it.
+- **Queue Owner AWS Account ID** (`queueOwnerAWSAccountId`): the 12-digit ID
of the account that owns the queue. The address is then built from the account,
the region and the queue name. Use it for a queue of another AWS account too.
+- An ARN in **Queue Name** (`queueNameOrArn`), which holds the account.
+
+=== Too Many Headers
+
+An SQS message takes at most 10 message attributes. The headers of the message
are sent with each part. When a message has more than 10 headers to send, the
first 10 are sent with each part, and `messageHeaderExceededLimit` sets what
happens to the others: `WARN` (the default) logs a warning for each one,
`WARN_ONCE` logs one warning for each part, `IGNORE` logs nothing, and `FAIL`
fails the message, which is then not sent.
+
+=== Encrypting the Queue
+
+`serverSideEncryptionEnabled` and `kmsMasterKeyId` set the queue to encrypt
its messages with a key of AWS KMS (SSE-KMS). They change the queue itself, not
only the messages of this Kamelet: when the integration starts, an existing
queue is updated, which needs the `sqs:SetQueueAttributes` permission, and a
queue that `autoCreateQueue` creates is created with the key.
`serverSideEncryptionEnabled` without `kmsMasterKeyId` has no effect.
+
+For example, to send to a queue without listing the queues, with each part
delayed by one minute:
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:aws-sqs-batch-sink"
+ parameters:
+ queueNameOrArn: "orders"
+ region: "eu-west-1"
+ queueURL: "https://sqs.eu-west-1.amazonaws.com/123456789012/orders"
+ batchSeparator: ","
+ delaySeconds: 60
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-sqs-fifo-sink-description.adoc
b/docs/modules/ROOT/partials/aws-sqs-fifo-sink-description.adoc
index d552e2e54..19c1a57e4 100644
--- a/docs/modules/ROOT/partials/aws-sqs-fifo-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sqs-fifo-sink-description.adoc
@@ -19,6 +19,34 @@ Only one of access key/secret key or default credentials
provider could be used
For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html[AWS
credentials documentation]
+=== Without Permission to List the Queues
+
+By default the Kamelet finds the address of the queue by listing the queues,
which needs the `sqs:ListQueues` permission. It does not list the queues when
it is given the address:
+
+- **Queue URL** (`queueURL`): the full address of the queue, as the SQS
console shows it.
+- **Queue Owner AWS Account ID** (`queueOwnerAWSAccountId`): the 12-digit ID
of the account that owns the queue. The address is then built from the account,
the region and the queue name. Use it for a queue of another AWS account too.
+- An ARN in **Queue Name** (`queueNameOrArn`), which holds the account.
+
+=== Too Many Headers
+
+An SQS message takes at most 10 message attributes. When a message has more
than 10 headers to send, the first 10 are sent, and
`messageHeaderExceededLimit` sets what happens to the others: `WARN` (the
default) logs a warning for each one, `WARN_ONCE` logs one warning, `IGNORE`
logs nothing, and `FAIL` fails the message, which is then not sent.
+
+=== Encrypting the Queue
+
+`serverSideEncryptionEnabled` and `kmsMasterKeyId` set the queue to encrypt
its messages with a key of AWS KMS (SSE-KMS). They change the queue itself, not
only the messages of this Kamelet: when the integration starts, an existing
queue is updated, which needs the `sqs:SetQueueAttributes` permission, and a
queue that `autoCreateQueue` creates is created with the key.
`serverSideEncryptionEnabled` without `kmsMasterKeyId` has no effect.
+
+For example, to send to a queue without listing the queues:
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:aws-sqs-fifo-sink"
+ parameters:
+ queueNameOrArn: "orders.fifo"
+ region: "eu-west-1"
+ queueURL: "https://sqs.eu-west-1.amazonaws.com/123456789012/orders.fifo"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-sqs-sink-description.adoc
b/docs/modules/ROOT/partials/aws-sqs-sink-description.adoc
index f37898271..4c2bb4bca 100644
--- a/docs/modules/ROOT/partials/aws-sqs-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sqs-sink-description.adoc
@@ -19,6 +19,40 @@ Only one of access key/secret key or default credentials
provider could be used
For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html[AWS
credentials documentation]
+=== Delaying the Messages
+
+`delaySeconds` keeps each message hidden for that many seconds after it is
sent, up to 900 (15 minutes), before a reader can receive it. Use it to process
a message later, such as a retry in a minute.
+
+=== Without Permission to List the Queues
+
+By default the Kamelet finds the address of the queue by listing the queues,
which needs the `sqs:ListQueues` permission. It does not list the queues when
it is given the address:
+
+- **Queue URL** (`queueURL`): the full address of the queue, as the SQS
console shows it.
+- **Queue Owner AWS Account ID** (`queueOwnerAWSAccountId`): the 12-digit ID
of the account that owns the queue. The address is then built from the account,
the region and the queue name. Use it for a queue of another AWS account too.
+- An ARN in **Queue Name** (`queueNameOrArn`), which holds the account.
+
+=== Too Many Headers
+
+An SQS message takes at most 10 message attributes. When a message has more
than 10 headers to send, the first 10 are sent, and
`messageHeaderExceededLimit` sets what happens to the others: `WARN` (the
default) logs a warning for each one, `WARN_ONCE` logs one warning, `IGNORE`
logs nothing, and `FAIL` fails the message, which is then not sent.
+
+=== Encrypting the Queue
+
+`serverSideEncryptionEnabled` and `kmsMasterKeyId` set the queue to encrypt
its messages with a key of AWS KMS (SSE-KMS). They change the queue itself, not
only the messages of this Kamelet: when the integration starts, an existing
queue is updated, which needs the `sqs:SetQueueAttributes` permission, and a
queue that `autoCreateQueue` creates is created with the key.
`serverSideEncryptionEnabled` without `kmsMasterKeyId` has no effect.
+
+For example, to send to a queue without listing the queues, with each message
delayed by one minute, and failing a message with too many headers:
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:aws-sqs-sink"
+ parameters:
+ queueNameOrArn: "orders"
+ region: "eu-west-1"
+ queueURL: "https://sqs.eu-west-1.amazonaws.com/123456789012/orders"
+ delaySeconds: 60
+ messageHeaderExceededLimit: "FAIL"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-sqs-source-description.adoc
b/docs/modules/ROOT/partials/aws-sqs-source-description.adoc
index 650a1b7b9..9fc725f73 100644
--- a/docs/modules/ROOT/partials/aws-sqs-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-sqs-source-description.adoc
@@ -49,6 +49,40 @@ messageAttributeNames: All
attributeNames: SentTimestamp
----
+=== Slow Messages and Parallel Processing
+
+A received message stays hidden from other readers for its visibility timeout,
30 seconds for a new queue. A message that takes longer to process is received
again, by this or another reader, and processed twice. Set
`extendMessageVisibility` to true, together with `visibilityTimeout`, to keep
each message hidden for as long as it is processed: every half of
`visibilityTimeout`, its visibility timeout is extended by `visibilityTimeout`
seconds.
+
+By default one thread polls the queue, and the messages are processed one
after the other. `concurrentConsumers` sets how many threads poll and process
at the same time.
+
+For example, to process up to five slow messages at a time, without receiving
any of them twice:
+
+[source,yaml]
+----
+queueNameOrArn: orders
+region: eu-west-1
+concurrentConsumers: 5
+visibilityTimeout: 60
+extendMessageVisibility: true
+----
+
+=== Without Permission to List the Queues
+
+By default the Kamelet finds the address of the queue by listing the queues,
which needs the `sqs:ListQueues` permission. It does not list the queues when
it is given the address:
+
+- **Queue URL** (`queueURL`): the full address of the queue, as the SQS
console shows it.
+- **Queue Owner AWS Account ID** (`queueOwnerAWSAccountId`): the 12-digit ID
of the account that owns the queue. The address is then built from the account,
the region and the queue name. Use it for a queue of another AWS account too.
+- An ARN in **Queue Name** (`queueNameOrArn`), which holds the account.
+
+For example, for the queue `orders` of the account `123456789012`:
+
+[source,yaml]
+----
+queueNameOrArn: orders
+region: eu-west-1
+queueOwnerAWSAccountId: "123456789012"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/kamelets/aws-sns-fifo-sink.kamelet.yaml
b/kamelets/aws-sns-fifo-sink.kamelet.yaml
index 0fac8c16e..43b8697c7 100644
--- a/kamelets/aws-sns-fifo-sink.kamelet.yaml
+++ b/kamelets/aws-sns-fifo-sink.kamelet.yaml
@@ -36,11 +36,11 @@ spec:
Use it to fan out ordered updates, such as the changes of one order or
one account, to several FIFO queues at once. For a standard topic use
`aws-sns-sink`.
- Input: the body becomes the text of the notification, as text. The
optional `group` header (or `ce-group`) is the message group ID, such as a
customer ID: the messages of one group are delivered in order. Without it, each
message is a group of its own. The optional `subject` header (or `ce-subject`)
is the subject. The other headers of the message, except the Camel ones, are
sent as SNS message attributes.
+ Input: the body becomes the text of the notification, as text. The
optional `group` header (or `ce-group`) is the message group ID, such as a
customer ID: the messages of one group are delivered in order. Without it, each
message is a group of its own. The optional `subject` header (or `ce-subject`)
is the subject; it takes precedence over the `subject` option. The other
headers of the message, except the Camel ones, are sent as SNS message
attributes.
Output: the message is unchanged. The `CamelAwsSnsMessageId` header
gives the ID of the published message.
- Good to know: the name of the topic ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SNS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the topic must allow. To reach
AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: the name of the topic ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SNS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the topic must allow. With
`messageStructure` set to `json`, the body is a JSON object with a text for
each kind of subscriber. `serverSideEncryptionEnabled` and `kmsMasterKeyId`
apply only to a topic that `autoCreateTopic` cr [...]
required:
- topicNameOrArn
- region
@@ -135,6 +135,27 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ subject:
+ title: Subject
+ description: The subject of every notification, which email
subscribers see as the subject line, and an SQS queue subscribed to the topic
gets in the `Subject` field of the notification. A `subject` or `ce-subject`
header on the message takes precedence. It is ASCII text without line breaks; a
subject longer than 100 characters is cut to 100. When it is not set, only the
messages with a header have a subject.
+ example: Order received
+ type: string
+ pattern: "^[ -~]+$"
+ messageStructure:
+ title: Message Structure
+ description: "Set it to `json` to send a different text to each kind
of subscriber. The body is then a JSON object with a `default` text for all
subscribers and, optionally, a text for each protocol, such as `{\"default\":
\"Order 42 received\", \"email\": \"Your order 42 was received.\"}`; SNS
rejects a body without `default`. When it is not set, the body is the text for
all subscribers."
+ type: string
+ enum: ["json"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: If true, a topic that `autoCreateTopic` creates is
created to encrypt its messages with the key of AWS KMS in `kmsMasterKeyId`,
which is then required. It does not change a topic that exists already.
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for a new topic to encrypt its
messages with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`,
an alias, such as `alias/aws/sns`, or its ARN. Only used when
`serverSideEncryptionEnabled` and `autoCreateTopic` are true and the topic does
not exist yet.
+ example: alias/aws/sns
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
@@ -144,7 +165,7 @@ spec:
type: string
subject:
title: Subject
- description: The subject of the notification. `ce-subject` works too.
+ description: The subject of the notification. `ce-subject` works
too. It takes precedence over the `subject` option.
type: string
dependencies:
- "camel:aws2-sns"
@@ -231,6 +252,10 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
otherwise:
steps:
- to:
@@ -253,3 +278,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git a/kamelets/aws-sns-sink.kamelet.yaml
b/kamelets/aws-sns-sink.kamelet.yaml
index e3730c0f1..4e36c24b2 100644
--- a/kamelets/aws-sns-sink.kamelet.yaml
+++ b/kamelets/aws-sns-sink.kamelet.yaml
@@ -37,11 +37,11 @@ spec:
Use it to fan out an event to many receivers at once, or to send
notifications such as alerts by email or SMS through a topic. For a FIFO topic
use `aws-sns-fifo-sink`; to send to a single queue use `aws-sqs-sink`.
- Input: the body becomes the text of the notification, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `subject` header (or `ce-subject`) is the
subject, which email subscribers see as the subject line. The other headers of
the message, except the Camel ones, are sent as SNS message attributes.
+ Input: the body becomes the text of the notification, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `subject` header (or `ce-subject`) is the
subject, which email subscribers see as the subject line; it takes precedence
over the `subject` option. The other headers of the message, except the Camel
ones, are sent as SNS message attributes.
Output: the message is unchanged. The `CamelAwsSnsMessageId` header
gives the ID of the published message.
- Good to know: to reach AWS through an HTTP proxy, set `proxyHost` and
`proxyPort`, and set `proxyProtocol` to `HTTP` for a proxy that takes plain
HTTP, such as Squid.
+ Good to know: with `messageStructure` set to `json`, the body is a JSON
object with a text for each kind of subscriber. `serverSideEncryptionEnabled`
and `kmsMasterKeyId` apply only to a topic that `autoCreateTopic` creates. To
reach AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
required:
- topicNameOrArn
- region
@@ -131,12 +131,33 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ subject:
+ title: Subject
+ description: The subject of every notification, which email
subscribers see as the subject line, and an SQS queue subscribed to the topic
gets in the `Subject` field of the notification. A `subject` or `ce-subject`
header on the message takes precedence. It is ASCII text without line breaks; a
subject longer than 100 characters is cut to 100. When it is not set, only the
messages with a header have a subject.
+ example: Order received
+ type: string
+ pattern: "^[ -~]+$"
+ messageStructure:
+ title: Message Structure
+ description: "Set it to `json` to send a different text to each kind
of subscriber. The body is then a JSON object with a `default` text for all
subscribers and, optionally, a text for each protocol, such as `{\"default\":
\"Order 42 received\", \"email\": \"Your order 42 was received.\"}`; SNS
rejects a body without `default`. When it is not set, the body is the text for
all subscribers."
+ type: string
+ enum: ["json"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: If true, a topic that `autoCreateTopic` creates is
created to encrypt its messages with the key of AWS KMS in `kmsMasterKeyId`,
which is then required. It does not change a topic that exists already.
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for a new topic to encrypt its
messages with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`,
an alias, such as `alias/aws/sns`, or its ARN. Only used when
`serverSideEncryptionEnabled` and `autoCreateTopic` are true and the topic does
not exist yet.
+ example: alias/aws/sns
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
subject:
title: Subject
- description: The subject of the notification, which email
subscribers see as the subject line. `ce-subject` works too.
+ description: The subject of the notification, which email
subscribers see as the subject line. `ce-subject` works too. It takes
precedence over the `subject` option.
type: string
dependencies:
- "camel:aws2-sns"
@@ -183,3 +204,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git a/kamelets/aws-sqs-batch-sink.kamelet.yaml
b/kamelets/aws-sqs-batch-sink.kamelet.yaml
index 0ca3a4ed7..336a5659b 100644
--- a/kamelets/aws-sqs-batch-sink.kamelet.yaml
+++ b/kamelets/aws-sqs-batch-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
response of SQS, and the `CamelAwsSqsSuccessfulMessageCount` and
`CamelAwsSqsFailedMessageCount` headers say how many parts were sent and how
many failed.
- Good to know: the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides `sqs:SendMessage`. To reach AWS through
an HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to
`HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: unless `queueURL`, `queueOwnerAWSAccountId` or an ARN in
`queueNameOrArn` gives the address of the queue, the queue is looked up by
listing the queues, which needs the `sqs:ListQueues` permission besides
`sqs:SendMessage`. `delaySeconds` keeps each message hidden for a while after
it is sent. `serverSideEncryptionEnabled` with `kmsMasterKeyId` changes the
queue itself when the integration starts, so that SQS encrypts its messages
with that KMS key. To reach AWS throug [...]
required:
- queueNameOrArn
- region
@@ -147,6 +147,41 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ delaySeconds:
+ title: Delay Seconds
+ description: How long SQS keeps each message hidden after it is sent,
before a reader can receive it, in seconds, from 0 to 900 (15 minutes). Use it
to process a message later, such as a retry in a minute. When it is not set,
the delay of the queue is used, which is 0 for a new queue.
+ example: 60
+ type: integer
+ minimum: "0"
+ maximum: "900"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning for each part, `IGNORE` sends 10 without a warning, and `FAIL` fails
the message, which is not sent. The 10 that are sent are the first 10 in the
order the headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
out:
headers:
@@ -187,3 +222,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ delaySeconds: "{{?delaySeconds}}"
+ messageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git a/kamelets/aws-sqs-fifo-sink.kamelet.yaml
b/kamelets/aws-sqs-fifo-sink.kamelet.yaml
index 4fc1cae03..5f256afa8 100644
--- a/kamelets/aws-sqs-fifo-sink.kamelet.yaml
+++ b/kamelets/aws-sqs-fifo-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the message is unchanged. The `CamelAwsSqsMessageId` header
gives the ID of the SQS message.
- Good to know: the name of the queue ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SQS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the queue must allow. The queue
is looked up by listing the queues, which needs the `sqs:ListQueues`
permission. To reach AWS through an HTTP proxy, set `proxyHost` and
`proxyPort`, and set `proxyProtocol` to `HTTP` for a prox [...]
+ Good to know: the name of the queue ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SQS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the queue must allow. Unless
`queueURL`, `queueOwnerAWSAccountId` or an ARN in `queueNameOrArn` gives the
address of the queue, the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides [...]
required:
- queueNameOrArn
- region
@@ -146,6 +146,34 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning, `IGNORE` sends 10 without a warning, and `FAIL` fails the message,
which is not sent. The 10 that are sent are the first 10 in the order the
headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
@@ -220,6 +248,11 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ messageHeaderExceededLimit:
"{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
otherwise:
steps:
- to:
@@ -244,3 +277,8 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ messageHeaderExceededLimit:
"{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git a/kamelets/aws-sqs-sink.kamelet.yaml
b/kamelets/aws-sqs-sink.kamelet.yaml
index c79025d27..5852ed760 100644
--- a/kamelets/aws-sqs-sink.kamelet.yaml
+++ b/kamelets/aws-sqs-sink.kamelet.yaml
@@ -41,7 +41,7 @@ spec:
Output: the message is unchanged. The `CamelAwsSqsMessageId` header
gives the ID of the SQS message.
- Good to know: the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides `sqs:SendMessage`. To reach a local
stand-in of AWS, use `overrideEndpoint` and `uriEndpointOverride`. To reach AWS
through an HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol`
to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: unless `queueURL`, `queueOwnerAWSAccountId` or an ARN in
`queueNameOrArn` gives the address of the queue, the queue is looked up by
listing the queues, which needs the `sqs:ListQueues` permission besides
`sqs:SendMessage`. `delaySeconds` keeps each message hidden for a while after
it is sent. `serverSideEncryptionEnabled` with `kmsMasterKeyId` changes the
queue itself when the integration starts, so that SQS encrypts its messages
with that KMS key. To reach a local st [...]
required:
- queueNameOrArn
- region
@@ -142,6 +142,41 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ delaySeconds:
+ title: Delay Seconds
+ description: How long SQS keeps each message hidden after it is sent,
before a reader can receive it, in seconds, from 0 to 900 (15 minutes). Use it
to process a message later, such as a retry in a minute. When it is not set,
the delay of the queue is used, which is 0 for a new queue.
+ example: 60
+ type: integer
+ minimum: "0"
+ maximum: "900"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning, `IGNORE` sends 10 without a warning, and `FAIL` fails the message,
which is not sent. The 10 that are sent are the first 10 in the order the
headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dependencies:
- "camel:aws2-sqs"
- "camel:kamelet"
@@ -169,3 +204,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ delaySeconds: "{{?delaySeconds}}"
+ messageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git a/kamelets/aws-sqs-source.kamelet.yaml
b/kamelets/aws-sqs-source.kamelet.yaml
index c3d629508..de06219c9 100644
--- a/kamelets/aws-sqs-source.kamelet.yaml
+++ b/kamelets/aws-sqs-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the body of the SQS message, as text. The headers
give the message ID (`CamelAwsSqsMessageId`), the receipt handle and the MD5
checksum of the body. The message attributes, which the sender sets on a
message, are read only when `messageAttributeNames` names them, and the system
attributes of SQS, such as `SentTimestamp`, only when `attributeNames` names
them. Each attribute that is read becomes a header of the same name, except a
message attribute whose name sta [...]
- Good to know: a message is deleted only after it has been processed
without error; a message that fails stays in the queue and is received again
when its visibility timeout ends. With `deleteAfterRead` set to false, the
messages stay in the queue and are received again and again. Without
`queueURL`, the queue is looked up by listing the queues, which needs the
`sqs:ListQueues` permission. To reach AWS through an HTTP proxy, set
`proxyHost` and `proxyPort`, and set `proxyProtocol` t [...]
+ Good to know: a message is deleted only after it has been processed
without error; a message that fails stays in the queue and is received again
when its visibility timeout ends. With `deleteAfterRead` set to false, the
messages stay in the queue and are received again and again. A message that
takes longer to process than the visibility timeout is received a second time,
by this or another reader: set `extendMessageVisibility` to keep it hidden for
as long as it is processed. By d [...]
required:
- queueNameOrArn
- region
@@ -172,6 +172,22 @@ spec:
description: "Which of the system attributes that SQS keeps for each
message to read: `All`, or a comma-separated list of names from `SenderId`,
`SentTimestamp`, `ApproximateReceiveCount`, `ApproximateFirstReceiveTimestamp`,
`SequenceNumber`, `MessageDeduplicationId`, `MessageGroupId`, `AWSTraceHeader`
and `DeadLetterQueueSourceArn`. A name that is not one of these is ignored,
with a warning in the log. Each attribute that is read becomes a header of the
same name, as text; the t [...]
example: SentTimestamp,ApproximateReceiveCount
type: string
+ concurrentConsumers:
+ title: Concurrent Consumers
+ description: How many threads poll the queue and process its messages
at the same time. When it is not set, one thread does, and the messages are
processed one after the other. With more, they are processed in parallel, in no
particular order.
+ example: 5
+ type: integer
+ minimum: "1"
+ extendMessageVisibility:
+ title: Extend Message Visibility
+ description: If true, a message is kept hidden from other readers for
as long as it is processed, so a message that takes longer than the visibility
timeout is not received a second time. It needs `visibilityTimeout`, without
which the integration does not start. Every half of `visibilityTimeout`, the
visibility timeout of the messages being processed is extended by
`visibilityTimeout` seconds; the credentials need the
`sqs:ChangeMessageVisibility` permission.
+ type: boolean
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -277,6 +293,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ concurrentConsumers: "{{?concurrentConsumers}}"
+ extendMessageVisibility: "{{?extendMessageVisibility}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
index fa37ce6e5..449dc50dd 100644
---
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
+++
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
@@ -56,7 +56,7 @@ public class KameletsCatalogTest {
@Test
void testGetKameletsDefinition() throws Exception {
Definition props = catalog.getKameletDefinition("aws-sqs-source");
- assertEquals(26, props.getProperties().keySet().size());
+ assertEquals(29, props.getProperties().keySet().size());
assertTrue(props.getProperties().containsKey("queueNameOrArn"));
assertTrue(props.getProperties().containsKey("deleteAfterRead"));
assertTrue(props.getProperties().containsKey("accessKey"));
@@ -76,6 +76,9 @@ public class KameletsCatalogTest {
assertTrue(props.getProperties().containsKey("messageAttributeNames"));
assertTrue(props.getProperties().containsKey("attributeNames"));
assertTrue(props.getProperties().containsKey("visibilityTimeout"));
+ assertTrue(props.getProperties().containsKey("concurrentConsumers"));
+
assertTrue(props.getProperties().containsKey("extendMessageVisibility"));
+
assertTrue(props.getProperties().containsKey("queueOwnerAWSAccountId"));
}
@Test
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 962a83ce4..43265e3ef 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -255,7 +255,7 @@
"supportLevel" : "Stable",
"title" : "AWS SNS FIFO Sink",
"summary" : "Publishes each message to an Amazon SNS FIFO topic, which
delivers the messages of a group in order, and without duplicates, to the SQS
FIFO queues subscribed to it.",
- "description" : "Publishes each message to an Amazon SNS FIFO topic, which
delivers the messages of a group in order, and without duplicates, to the SQS
FIFO queues subscribed to it.\n\nUse it to fan out ordered updates, such as the
changes of one order or one account, to several FIFO queues at once. For a
standard topic use `aws-sns-sink`.\n\nInput: the body becomes the text of the
notification, as text. The optional `group` header (or `ce-group`) is the
message group ID, such as a [...]
+ "description" : "Publishes each message to an Amazon SNS FIFO topic, which
delivers the messages of a group in order, and without duplicates, to the SQS
FIFO queues subscribed to it.\n\nUse it to fan out ordered updates, such as the
changes of one order or one account, to several FIFO queues at once. For a
standard topic use `aws-sns-sink`.\n\nInput: the body becomes the text of the
notification, as text. The optional `group` header (or `ce-group`) is the
message group ID, such as a [...]
"group" : "AWS SNS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -265,7 +265,7 @@
"supportLevel" : "Stable",
"title" : "AWS SNS Sink",
"summary" : "Publishes each message to an Amazon SNS topic, which delivers
it to every subscriber of the topic, such as SQS queues, Lambda functions, HTTP
addresses, email addresses or phone numbers.",
- "description" : "Publishes each message to an Amazon SNS topic, which
delivers it to every subscriber of the topic, such as SQS queues, Lambda
functions, HTTP addresses, email addresses or phone numbers.\n\nUse it to fan
out an event to many receivers at once, or to send notifications such as alerts
by email or SMS through a topic. For a FIFO topic use `aws-sns-fifo-sink`; to
send to a single queue use `aws-sqs-sink`.\n\nInput: the body becomes the text
of the notification, as text. [...]
+ "description" : "Publishes each message to an Amazon SNS topic, which
delivers it to every subscriber of the topic, such as SQS queues, Lambda
functions, HTTP addresses, email addresses or phone numbers.\n\nUse it to fan
out an event to many receivers at once, or to send notifications such as alerts
by email or SMS through a topic. For a FIFO topic use `aws-sns-fifo-sink`; to
send to a single queue use `aws-sqs-sink`.\n\nInput: the body becomes the text
of the notification, as text. [...]
"group" : "AWS SNS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -275,7 +275,7 @@
"supportLevel" : "Stable",
"title" : "AWS SQS Batch Sink",
"summary" : "Splits the text of each message at a separator and sends the
parts as separate SQS messages to an Amazon SQS queue, in one batch call.",
- "description" : "Splits the text of each message at a separator and sends
the parts as separate SQS messages to an Amazon SQS queue, in one batch
call.\n\nUse it to send many small items that arrive together, such as a
comma-separated list of IDs, as one SQS message each, with fewer calls to AWS.
To send each message as one SQS message, use `aws-sqs-sink`.\n\nInput: the body
is text that `batchSeparator` splits, such as `one,two,three` with a separator
of `,`; each part becomes the b [...]
+ "description" : "Splits the text of each message at a separator and sends
the parts as separate SQS messages to an Amazon SQS queue, in one batch
call.\n\nUse it to send many small items that arrive together, such as a
comma-separated list of IDs, as one SQS message each, with fewer calls to AWS.
To send each message as one SQS message, use `aws-sqs-sink`.\n\nInput: the body
is text that `batchSeparator` splits, such as `one,two,three` with a separator
of `,`; each part becomes the b [...]
"group" : "AWS SQS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -285,7 +285,7 @@
"supportLevel" : "Stable",
"title" : "AWS SQS FIFO Sink",
"summary" : "Sends each message to an Amazon SQS FIFO queue, which
delivers the messages of a group in the order they were sent and drops
duplicates.",
- "description" : "Sends each message to an Amazon SQS FIFO queue, which
delivers the messages of a group in the order they were sent and drops
duplicates.\n\nUse it to queue messages whose order matters, such as the
updates of one order or one account, which must be processed in the order they
happened. For a standard queue use `aws-sqs-sink`.\n\nInput: the body becomes
the body of the SQS message, as text. The optional `group` header (or
`ce-group`) is the message group ID, such as a [...]
+ "description" : "Sends each message to an Amazon SQS FIFO queue, which
delivers the messages of a group in the order they were sent and drops
duplicates.\n\nUse it to queue messages whose order matters, such as the
updates of one order or one account, which must be processed in the order they
happened. For a standard queue use `aws-sqs-sink`.\n\nInput: the body becomes
the body of the SQS message, as text. The optional `group` header (or
`ce-group`) is the message group ID, such as a [...]
"group" : "AWS SQS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -295,7 +295,7 @@
"supportLevel" : "Stable",
"title" : "AWS SQS Sink",
"summary" : "Sends each message as an SQS message to an Amazon SQS (Simple
Queue Service) queue.",
- "description" : "Sends each message as an SQS message to an Amazon SQS
(Simple Queue Service) queue.\n\nUse it to hand work to other systems that read
the queue, such as orders to process or jobs to run, and to buffer it when they
are slow or down. For a FIFO queue use `aws-sqs-fifo-sink`, to send many parts
in one call use `aws-sqs-batch-sink`, and to notify many receivers at once use
`aws-sns-sink`.\n\nInput: the body becomes the body of the SQS message, as
text; bytes and streams [...]
+ "description" : "Sends each message as an SQS message to an Amazon SQS
(Simple Queue Service) queue.\n\nUse it to hand work to other systems that read
the queue, such as orders to process or jobs to run, and to buffer it when they
are slow or down. For a FIFO queue use `aws-sqs-fifo-sink`, to send many parts
in one call use `aws-sqs-batch-sink`, and to notify many receivers at once use
`aws-sns-sink`.\n\nInput: the body becomes the body of the SQS message, as
text; bytes and streams [...]
"group" : "AWS SQS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -305,7 +305,7 @@
"supportLevel" : "Stable",
"title" : "AWS SQS Source",
"summary" : "Receives the messages of an Amazon SQS queue, one message
each, and by default deletes each message from the queue once it has been
processed.",
- "description" : "Receives the messages of an Amazon SQS queue, one message
each, and by default deletes each message from the queue once it has been
processed.\n\nUse it to process work that other systems put in a queue, such as
orders, jobs or notifications, or the events that SNS, S3 or EventBridge send
to a queue. For the events of an S3 bucket, `aws-s3-event-based-source` can
also fetch the files.\n\nOutput: the body is the body of the SQS message, as
text. The headers give the m [...]
+ "description" : "Receives the messages of an Amazon SQS queue, one message
each, and by default deletes each message from the queue once it has been
processed.\n\nUse it to process work that other systems put in a queue, such as
orders, jobs or notifications, or the events that SNS, S3 or EventBridge send
to a queue. For the events of an S3 bucket, `aws-s3-event-based-source` can
also fetch the files.\n\nOutput: the body is the body of the SQS message, as
text. The headers give the m [...]
"group" : "AWS SQS",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sns-fifo-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sns-fifo-sink.kamelet.yaml
index 0fac8c16e..43b8697c7 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sns-fifo-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sns-fifo-sink.kamelet.yaml
@@ -36,11 +36,11 @@ spec:
Use it to fan out ordered updates, such as the changes of one order or
one account, to several FIFO queues at once. For a standard topic use
`aws-sns-sink`.
- Input: the body becomes the text of the notification, as text. The
optional `group` header (or `ce-group`) is the message group ID, such as a
customer ID: the messages of one group are delivered in order. Without it, each
message is a group of its own. The optional `subject` header (or `ce-subject`)
is the subject. The other headers of the message, except the Camel ones, are
sent as SNS message attributes.
+ Input: the body becomes the text of the notification, as text. The
optional `group` header (or `ce-group`) is the message group ID, such as a
customer ID: the messages of one group are delivered in order. Without it, each
message is a group of its own. The optional `subject` header (or `ce-subject`)
is the subject; it takes precedence over the `subject` option. The other
headers of the message, except the Camel ones, are sent as SNS message
attributes.
Output: the message is unchanged. The `CamelAwsSnsMessageId` header
gives the ID of the published message.
- Good to know: the name of the topic ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SNS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the topic must allow. To reach
AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: the name of the topic ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SNS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the topic must allow. With
`messageStructure` set to `json`, the body is a JSON object with a text for
each kind of subscriber. `serverSideEncryptionEnabled` and `kmsMasterKeyId`
apply only to a topic that `autoCreateTopic` cr [...]
required:
- topicNameOrArn
- region
@@ -135,6 +135,27 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ subject:
+ title: Subject
+ description: The subject of every notification, which email
subscribers see as the subject line, and an SQS queue subscribed to the topic
gets in the `Subject` field of the notification. A `subject` or `ce-subject`
header on the message takes precedence. It is ASCII text without line breaks; a
subject longer than 100 characters is cut to 100. When it is not set, only the
messages with a header have a subject.
+ example: Order received
+ type: string
+ pattern: "^[ -~]+$"
+ messageStructure:
+ title: Message Structure
+ description: "Set it to `json` to send a different text to each kind
of subscriber. The body is then a JSON object with a `default` text for all
subscribers and, optionally, a text for each protocol, such as `{\"default\":
\"Order 42 received\", \"email\": \"Your order 42 was received.\"}`; SNS
rejects a body without `default`. When it is not set, the body is the text for
all subscribers."
+ type: string
+ enum: ["json"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: If true, a topic that `autoCreateTopic` creates is
created to encrypt its messages with the key of AWS KMS in `kmsMasterKeyId`,
which is then required. It does not change a topic that exists already.
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for a new topic to encrypt its
messages with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`,
an alias, such as `alias/aws/sns`, or its ARN. Only used when
`serverSideEncryptionEnabled` and `autoCreateTopic` are true and the topic does
not exist yet.
+ example: alias/aws/sns
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
@@ -144,7 +165,7 @@ spec:
type: string
subject:
title: Subject
- description: The subject of the notification. `ce-subject` works too.
+ description: The subject of the notification. `ce-subject` works
too. It takes precedence over the `subject` option.
type: string
dependencies:
- "camel:aws2-sns"
@@ -231,6 +252,10 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
otherwise:
steps:
- to:
@@ -253,3 +278,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sns-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sns-sink.kamelet.yaml
index e3730c0f1..4e36c24b2 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sns-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sns-sink.kamelet.yaml
@@ -37,11 +37,11 @@ spec:
Use it to fan out an event to many receivers at once, or to send
notifications such as alerts by email or SMS through a topic. For a FIFO topic
use `aws-sns-fifo-sink`; to send to a single queue use `aws-sqs-sink`.
- Input: the body becomes the text of the notification, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `subject` header (or `ce-subject`) is the
subject, which email subscribers see as the subject line. The other headers of
the message, except the Camel ones, are sent as SNS message attributes.
+ Input: the body becomes the text of the notification, as text. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional `subject` header (or `ce-subject`) is the
subject, which email subscribers see as the subject line; it takes precedence
over the `subject` option. The other headers of the message, except the Camel
ones, are sent as SNS message attributes.
Output: the message is unchanged. The `CamelAwsSnsMessageId` header
gives the ID of the published message.
- Good to know: to reach AWS through an HTTP proxy, set `proxyHost` and
`proxyPort`, and set `proxyProtocol` to `HTTP` for a proxy that takes plain
HTTP, such as Squid.
+ Good to know: with `messageStructure` set to `json`, the body is a JSON
object with a text for each kind of subscriber. `serverSideEncryptionEnabled`
and `kmsMasterKeyId` apply only to a topic that `autoCreateTopic` creates. To
reach AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
required:
- topicNameOrArn
- region
@@ -131,12 +131,33 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ subject:
+ title: Subject
+ description: The subject of every notification, which email
subscribers see as the subject line, and an SQS queue subscribed to the topic
gets in the `Subject` field of the notification. A `subject` or `ce-subject`
header on the message takes precedence. It is ASCII text without line breaks; a
subject longer than 100 characters is cut to 100. When it is not set, only the
messages with a header have a subject.
+ example: Order received
+ type: string
+ pattern: "^[ -~]+$"
+ messageStructure:
+ title: Message Structure
+ description: "Set it to `json` to send a different text to each kind
of subscriber. The body is then a JSON object with a `default` text for all
subscribers and, optionally, a text for each protocol, such as `{\"default\":
\"Order 42 received\", \"email\": \"Your order 42 was received.\"}`; SNS
rejects a body without `default`. When it is not set, the body is the text for
all subscribers."
+ type: string
+ enum: ["json"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: If true, a topic that `autoCreateTopic` creates is
created to encrypt its messages with the key of AWS KMS in `kmsMasterKeyId`,
which is then required. It does not change a topic that exists already.
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for a new topic to encrypt its
messages with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`,
an alias, such as `alias/aws/sns`, or its ARN. Only used when
`serverSideEncryptionEnabled` and `autoCreateTopic` are true and the topic does
not exist yet.
+ example: alias/aws/sns
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
subject:
title: Subject
- description: The subject of the notification, which email
subscribers see as the subject line. `ce-subject` works too.
+ description: The subject of the notification, which email
subscribers see as the subject line. `ce-subject` works too. It takes
precedence over the `subject` option.
type: string
dependencies:
- "camel:aws2-sns"
@@ -183,3 +204,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ subject: "RAW({{?subject}})"
+ messageStructure: "{{?messageStructure}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-batch-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-batch-sink.kamelet.yaml
index 0ca3a4ed7..336a5659b 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-batch-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-batch-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
response of SQS, and the `CamelAwsSqsSuccessfulMessageCount` and
`CamelAwsSqsFailedMessageCount` headers say how many parts were sent and how
many failed.
- Good to know: the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides `sqs:SendMessage`. To reach AWS through
an HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to
`HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: unless `queueURL`, `queueOwnerAWSAccountId` or an ARN in
`queueNameOrArn` gives the address of the queue, the queue is looked up by
listing the queues, which needs the `sqs:ListQueues` permission besides
`sqs:SendMessage`. `delaySeconds` keeps each message hidden for a while after
it is sent. `serverSideEncryptionEnabled` with `kmsMasterKeyId` changes the
queue itself when the integration starts, so that SQS encrypts its messages
with that KMS key. To reach AWS throug [...]
required:
- queueNameOrArn
- region
@@ -147,6 +147,41 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ delaySeconds:
+ title: Delay Seconds
+ description: How long SQS keeps each message hidden after it is sent,
before a reader can receive it, in seconds, from 0 to 900 (15 minutes). Use it
to process a message later, such as a retry in a minute. When it is not set,
the delay of the queue is used, which is 0 for a new queue.
+ example: 60
+ type: integer
+ minimum: "0"
+ maximum: "900"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning for each part, `IGNORE` sends 10 without a warning, and `FAIL` fails
the message, which is not sent. The 10 that are sent are the first 10 in the
order the headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
out:
headers:
@@ -187,3 +222,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ delaySeconds: "{{?delaySeconds}}"
+ messageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-fifo-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-fifo-sink.kamelet.yaml
index 4fc1cae03..5f256afa8 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-fifo-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-fifo-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the message is unchanged. The `CamelAwsSqsMessageId` header
gives the ID of the SQS message.
- Good to know: the name of the queue ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SQS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the queue must allow. The queue
is looked up by listing the queues, which needs the `sqs:ListQueues`
permission. To reach AWS through an HTTP proxy, set `proxyHost` and
`proxyPort`, and set `proxyProtocol` to `HTTP` for a prox [...]
+ Good to know: the name of the queue ends in `.fifo`. With
`contentBasedDeduplication` false, the default, each message gets a unique
deduplication ID, so SQS drops only a message sent twice by a retry; set it to
true to drop messages with the same body, which the queue must allow. Unless
`queueURL`, `queueOwnerAWSAccountId` or an ARN in `queueNameOrArn` gives the
address of the queue, the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides [...]
required:
- queueNameOrArn
- region
@@ -146,6 +146,34 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning, `IGNORE` sends 10 without a warning, and `FAIL` fails the message,
which is not sent. The 10 that are sent are the first 10 in the order the
headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dataTypes:
in:
headers:
@@ -220,6 +248,11 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ messageHeaderExceededLimit:
"{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
otherwise:
steps:
- to:
@@ -244,3 +277,8 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ messageHeaderExceededLimit:
"{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-sink.kamelet.yaml
index c79025d27..5852ed760 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-sink.kamelet.yaml
@@ -41,7 +41,7 @@ spec:
Output: the message is unchanged. The `CamelAwsSqsMessageId` header
gives the ID of the SQS message.
- Good to know: the queue is looked up by listing the queues, which needs
the `sqs:ListQueues` permission besides `sqs:SendMessage`. To reach a local
stand-in of AWS, use `overrideEndpoint` and `uriEndpointOverride`. To reach AWS
through an HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol`
to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: unless `queueURL`, `queueOwnerAWSAccountId` or an ARN in
`queueNameOrArn` gives the address of the queue, the queue is looked up by
listing the queues, which needs the `sqs:ListQueues` permission besides
`sqs:SendMessage`. `delaySeconds` keeps each message hidden for a while after
it is sent. `serverSideEncryptionEnabled` with `kmsMasterKeyId` changes the
queue itself when the integration starts, so that SQS encrypts its messages
with that KMS key. To reach a local st [...]
required:
- queueNameOrArn
- region
@@ -142,6 +142,41 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ queueURL:
+ title: Queue URL
+ description: The full address of the queue, as the SQS console shows
it. When it is set, the messages are sent to it, and the queue is not looked up
by listing the queues, which needs the `sqs:ListQueues` permission.
+ example: https://sqs.eu-west-1.amazonaws.com/123456789012/orders
+ type: string
+ pattern: "^https?://[^\\s]+$"
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
+ delaySeconds:
+ title: Delay Seconds
+ description: How long SQS keeps each message hidden after it is sent,
before a reader can receive it, in seconds, from 0 to 900 (15 minutes). Use it
to process a message later, such as a retry in a minute. When it is not set,
the delay of the queue is used, which is 0 for a new queue.
+ example: 60
+ type: integer
+ minimum: "0"
+ maximum: "900"
+ messageHeaderExceededLimit:
+ title: Too Many Headers
+ description: "What to do when a message has more than 10 headers to
send as message attributes, the most that SQS takes: `WARN` sends 10 of them
and logs a warning for each other one, `WARN_ONCE` sends 10 and logs one
warning, `IGNORE` sends 10 without a warning, and `FAIL` fails the message,
which is not sent. The 10 that are sent are the first 10 in the order the
headers were set on the message. When it is not set, it is `WARN`."
+ example: FAIL
+ type: string
+ enum: ["WARN", "WARN_ONCE", "IGNORE", "FAIL"]
+ serverSideEncryptionEnabled:
+ title: Server-Side Encryption
+ description: "If true, the queue is set to encrypt its messages with
the key of AWS KMS in `kmsMasterKeyId` (SSE-KMS). It changes the queue itself:
when the integration starts, an existing queue is updated, which needs the
`sqs:SetQueueAttributes` permission, and a queue that `autoCreateQueue` creates
is created with it. Without `kmsMasterKeyId`, it has no effect."
+ type: boolean
+ kmsMasterKeyId:
+ title: KMS Key ID
+ description: The key of AWS KMS for the queue to encrypt its messages
with, as its key ID, such as `1234abcd-12ab-34cd-56ef-1234567890ab`, an alias,
such as `alias/aws/sqs`, or its ARN. Only used when
`serverSideEncryptionEnabled` is true.
+ example: alias/aws/sqs
+ type: string
+ pattern: "^[A-Za-z0-9:/_-]+$"
dependencies:
- "camel:aws2-sqs"
- "camel:kamelet"
@@ -169,3 +204,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ queueUrl: "{{?queueURL}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
+ delaySeconds: "{{?delaySeconds}}"
+ messageHeaderExceededLimit: "{{?messageHeaderExceededLimit}}"
+ serverSideEncryptionEnabled: "{{?serverSideEncryptionEnabled}}"
+ kmsMasterKeyId: "{{?kmsMasterKeyId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-source.kamelet.yaml
index c3d629508..de06219c9 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-sqs-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the body of the SQS message, as text. The headers
give the message ID (`CamelAwsSqsMessageId`), the receipt handle and the MD5
checksum of the body. The message attributes, which the sender sets on a
message, are read only when `messageAttributeNames` names them, and the system
attributes of SQS, such as `SentTimestamp`, only when `attributeNames` names
them. Each attribute that is read becomes a header of the same name, except a
message attribute whose name sta [...]
- Good to know: a message is deleted only after it has been processed
without error; a message that fails stays in the queue and is received again
when its visibility timeout ends. With `deleteAfterRead` set to false, the
messages stay in the queue and are received again and again. Without
`queueURL`, the queue is looked up by listing the queues, which needs the
`sqs:ListQueues` permission. To reach AWS through an HTTP proxy, set
`proxyHost` and `proxyPort`, and set `proxyProtocol` t [...]
+ Good to know: a message is deleted only after it has been processed
without error; a message that fails stays in the queue and is received again
when its visibility timeout ends. With `deleteAfterRead` set to false, the
messages stay in the queue and are received again and again. A message that
takes longer to process than the visibility timeout is received a second time,
by this or another reader: set `extendMessageVisibility` to keep it hidden for
as long as it is processed. By d [...]
required:
- queueNameOrArn
- region
@@ -172,6 +172,22 @@ spec:
description: "Which of the system attributes that SQS keeps for each
message to read: `All`, or a comma-separated list of names from `SenderId`,
`SentTimestamp`, `ApproximateReceiveCount`, `ApproximateFirstReceiveTimestamp`,
`SequenceNumber`, `MessageDeduplicationId`, `MessageGroupId`, `AWSTraceHeader`
and `DeadLetterQueueSourceArn`. A name that is not one of these is ignored,
with a warning in the log. Each attribute that is read becomes a header of the
same name, as text; the t [...]
example: SentTimestamp,ApproximateReceiveCount
type: string
+ concurrentConsumers:
+ title: Concurrent Consumers
+ description: How many threads poll the queue and process its messages
at the same time. When it is not set, one thread does, and the messages are
processed one after the other. With more, they are processed in parallel, in no
particular order.
+ example: 5
+ type: integer
+ minimum: "1"
+ extendMessageVisibility:
+ title: Extend Message Visibility
+ description: If true, a message is kept hidden from other readers for
as long as it is processed, so a message that takes longer than the visibility
timeout is not received a second time. It needs `visibilityTimeout`, without
which the integration does not start. Every half of `visibilityTimeout`, the
visibility timeout of the messages being processed is extended by
`visibilityTimeout` seconds; the credentials need the
`sqs:ChangeMessageVisibility` permission.
+ type: boolean
+ queueOwnerAWSAccountId:
+ title: Queue Owner AWS Account ID
+ description: The 12-digit ID of the AWS account that owns the queue,
such as `123456789012`. When it is set, the address of the queue is built from
the account, `region` and the queue name, and the queue is not looked up by
listing the queues. Use it for a queue of another AWS account, or when the
credentials may not list the queues. An ARN in `queueNameOrArn` gives the
account too.
+ example: "123456789012"
+ type: string
+ pattern: "^[0-9]{12}$"
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -277,6 +293,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ concurrentConsumers: "{{?concurrentConsumers}}"
+ extendMessageVisibility: "{{?extendMessageVisibility}}"
+ queueOwnerAWSAccountId: "{{?queueOwnerAWSAccountId}}"
steps:
- to:
uri: kamelet:sink