This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 59ae6c15d chore: the S3 sink can upload in parts and refuse to replace
a file, the S3 event source can use long polling, the SES sink takes its
recipients as options (#3170) (#3231)
59ae6c15d is described below
commit 59ae6c15d6d80de0facf7593122dd80e91f2e061
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:37:02 2026 +0100
chore: the S3 sink can upload in parts and refuse to replace a file, the S3
event source can use long polling, the SES sink takes its recipients as options
(#3170) (#3231)
aws-s3-sink:
- multiPartUpload and partSize: a file of partSize bytes or more is
uploaded as an S3 multipart upload, so files over the 5 GB limit of
one upload can be stored.
- conditionalWritesEnabled: the upload of a key that exists already
fails with PreconditionFailed (412) instead of replacing the file.
aws-s3-event-based-source: maxMessagesPerPoll, waitTimeSeconds (long
polling) and visibilityTimeout on its SQS consumer.
aws-ses-sink: to, cc, bcc, subject, replyToAddresses, returnPath and
configurationSet as options, and overrideEndpoint and
uriEndpointOverride to reach a local stand-in of AWS. The to, subject
and reply-to-addresses headers take precedence over the options; with
the to option set, the to header is no longer required. The fallback
of the subject to the exchange ID is skipped when the subject option
is set. The subject and the addresses go into the URI inside RAW(), so
that a +, a % or an & is kept as written.
All new options are optional and have no default: with them unset,
the endpoint URIs are the same as before. The descriptions and the
doc partials have examples of the new options.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../aws-s3-event-based-source-description.adoc | 30 ++++++++
.../ROOT/partials/aws-s3-sink-description.adoc | 32 +++++++++
.../ROOT/partials/aws-ses-sink-description.adoc | 61 +++++++++++++++-
kamelets/aws-s3-event-based-source.kamelet.yaml | 25 ++++++-
kamelets/aws-s3-sink.kamelet.yaml | 17 ++++-
kamelets/aws-ses-sink.kamelet.yaml | 81 ++++++++++++++++++----
.../main/resources/META-INF/kamelets-index.json | 8 +--
.../aws-s3-event-based-source.kamelet.yaml | 25 ++++++-
.../resources/kamelets/aws-s3-sink.kamelet.yaml | 17 ++++-
.../resources/kamelets/aws-ses-sink.kamelet.yaml | 81 ++++++++++++++++++----
10 files changed, 341 insertions(+), 36 deletions(-)
diff --git
a/docs/modules/ROOT/partials/aws-s3-event-based-source-description.adoc
b/docs/modules/ROOT/partials/aws-s3-event-based-source-description.adoc
index 74b99e83d..47fe0f15a 100644
--- a/docs/modules/ROOT/partials/aws-s3-event-based-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-s3-event-based-source-description.adoc
@@ -41,6 +41,36 @@ On Kubernetes, with an IAM role given to the pod through
IRSA or EKS Pod Identit
The role needs the permission to receive and delete the messages of the queue
and, with `getObject`, to read the files of the bucket.
+=== Example with Long Polling and a Longer Visibility Timeout
+
+By default, the Kamelet asks the queue for events every `delay` milliseconds,
and each request returns at once, even when
+the queue is empty. With **waitTimeSeconds**, a request waits up to that many
seconds for an event to arrive, so an
+empty queue costs far fewer requests, and a new event is still received as
soon as it arrives. **maxMessagesPerPoll**
+is the most events received in one request (1 to 10; 10 when it is not set).
**visibilityTimeout** is how long a
+received event stays hidden from other readers: set it longer than processing
one event takes, such as fetching a
+large file with `getObject`, or the event is received a second time while it
is still being processed. An event that
+fails stays in the queue and is received again when this time ends.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:aws-s3-event-based-source"
+ parameters:
+ queueNameOrArn: "s3-events"
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ getObject: true
+ waitTimeSeconds: 20
+ maxMessagesPerPoll: 5
+ visibilityTimeout: 300
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+When they are not set, the wait time and the visibility timeout of the queue
are used.
+
=== Required Setup
To use this Kamelet you'll need to set up Eventbridge on your bucket and
subscribe Eventbridge bus to an SQS Queue.
diff --git a/docs/modules/ROOT/partials/aws-s3-sink-description.adoc
b/docs/modules/ROOT/partials/aws-s3-sink-description.adoc
index 718b75939..eb07326b4 100644
--- a/docs/modules/ROOT/partials/aws-s3-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-s3-sink-description.adoc
@@ -60,6 +60,38 @@ to decrypt with it (`kms:Decrypt`). Without **awsKMSKeyId**,
**useAwsKMS** has n
S3, such as `STANDARD_IA`, `INTELLIGENT_TIERING` or `GLACIER_IR`; a file in
`GLACIER` or `DEEP_ARCHIVE` must be restored
before it can be read. A `CamelAwsS3StorageClass` header on the message takes
precedence over **storageClass**.
+=== Example with Multipart Uploads and Conditional Writes
+
+S3 takes at most 5 GB in one upload. With **multiPartUpload** set to `true`, a
file of **partSize** bytes or more is
+uploaded in parts, one request per part, and S3 joins them into one file;
smaller files are still uploaded in one
+request. **partSize** is 25 MB (`26214400`) when it is not set. S3 takes parts
of at least 5 MB (`5242880`), except the
+last one, and at most 10,000 parts per file, so for a file of 500 GB the parts
must be at least 50 MB. An upload that fails is
+aborted, so that S3 discards the parts sent so far.
+
+With **conditionalWritesEnabled** set to `true`, a file is stored only when
its key does not exist yet in the bucket.
+The upload of an existing key fails with `PreconditionFailed` (status 412),
and the message fails; the file in the
+bucket is not changed. Use it when a file must never be replaced, such as
received documents kept as they came.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "file:/data/outbox"
+ steps:
+ - setHeader:
+ name: file
+ simple: "archive/$\{header.CamelFileName}"
+ - to:
+ uri: "kamelet:aws-s3-sink"
+ parameters:
+ bucketNameOrArn: "my-bucket"
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ multiPartUpload: true
+ partSize: 104857600
+ conditionalWritesEnabled: true
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-ses-sink-description.adoc
b/docs/modules/ROOT/partials/aws-ses-sink-description.adoc
index a160ff0a3..f1a86f4a1 100644
--- a/docs/modules/ROOT/partials/aws-ses-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-ses-sink-description.adoc
@@ -21,16 +21,73 @@ For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/de
=== Headers
-The following header is required:
+The following header is required, unless the `to` option is set:
- `to` / `ce-to` to provide a comma-separated list of email addresses.
In the headers, you can optionally set the following properties:
-- `subject` / `ce-subject` property to specify the email subject. If you do
not set this property in the header, the Kamelet uses a generated unique ID for
the email subject.
+- `subject` / `ce-subject` property to specify the email subject. If you do
not set this property in the header, nor the `subject` option, the Kamelet uses
a generated unique ID for the email subject.
- `reply-to-addresses` / `ce-reply-to-addresses` to provide a comma-separated
list of reply-to email addresses.
+Each of these headers takes precedence over the option of the same purpose:
`to`, `subject` and `replyToAddresses`.
+
+=== Example with Fixed Recipients and a Configuration Set
+
+When every email goes to the same people, such as the alerts of a job, set the
recipients and the subject as options,
+and the messages need no headers. **cc** and **bcc** send a copy, and a blind
copy, of every email. **returnPath** is the
+address that bounces and complaints go to; it must be verified in SES, as
**from** is. **configurationSet** names an SES
+configuration set, which can publish the delivery, bounce, open and click
events of the emails to CloudWatch, SNS or
+Firehose; it must exist in SES in the region, or every email fails with
`ConfigurationSetDoesNotExist`.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "job-failures"
+ bootstrapServers: "kafka.example.com:9092"
+ saslAuthType: "NONE"
+ steps:
+ - to:
+ uri: "kamelet:aws-ses-sink"
+ parameters:
+ from: "[email protected]"
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ to: "[email protected],[email protected]"
+ cc: "[email protected]"
+ bcc: "[email protected]"
+ subject: "Nightly job failed"
+ replyToAddresses: "[email protected]"
+ returnPath: "[email protected]"
+ configurationSet: "alerts"
+----
+
+The addresses are plain addresses, separated by commas, such as
`[email protected],[email protected]`. A `to`, `subject`
+or `reply-to-addresses` header on a message still takes precedence over the
option.
+
+=== Example with a Local Stand-in of AWS
+
+To test without AWS, point the Kamelet to a local stand-in that has SES, such
as LocalStack or Floci, with
+**overrideEndpoint** and **uriEndpointOverride**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- to:
+ uri: "kamelet:aws-ses-sink"
+ parameters:
+ from: "[email protected]"
+ region: "us-east-1"
+ accessKey: "test"
+ secretKey: "test"
+ to: "[email protected]"
+ overrideEndpoint: true
+ uriEndpointOverride: "http://localhost:4566"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/kamelets/aws-s3-event-based-source.kamelet.yaml
b/kamelets/aws-s3-event-based-source.kamelet.yaml
index a26cfa578..4db8b9cdb 100644
--- a/kamelets/aws-s3-event-based-source.kamelet.yaml
+++ b/kamelets/aws-s3-event-based-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the EventBridge event, as JSON text: `detail-type`
says what happened (such as `Object Created` or `Object Deleted`),
`detail.bucket.name` is the bucket and `detail.object.key` the key of the file.
With `getObject` set to true, the event of a file uploaded with a simple upload
(PutObject) carries the content of the file instead, as a stream, with the
`CamelAwsS3ProducedBucketName` and `CamelAwsS3ProducedKey` headers; the other
events then come as parsed JSON (a [...]
- Good to know: it needs a setup in AWS. In the properties of the bucket,
turn on sending events to Amazon EventBridge; then create an EventBridge rule
for the events of source `aws.s3` and that bucket, with the queue as its
target. Each event is deleted from the queue once it has been processed, unless
`deleteAfterRead` is false. The files stay in the bucket. `getObject` skips
files created by a multipart upload or a copy, and never connects through
`uriEndpointOverride`. It logs in [...]
+ Good to know: it needs a setup in AWS. In the properties of the bucket,
turn on sending events to Amazon EventBridge; then create an EventBridge rule
for the events of source `aws.s3` and that bucket, with the queue as its
target. Each event is deleted from the queue once it has been processed, unless
`deleteAfterRead` is false; an event that fails stays in the queue and is
received again when its visibility timeout (`visibilityTimeout`) ends. The
files stay in the bucket. `getObje [...]
required:
- queueNameOrArn
- region
@@ -176,6 +176,26 @@ spec:
description: If true, it polls again right away when the last poll
received messages, instead of waiting for `delay`.
type: boolean
default: false
+ maxMessagesPerPoll:
+ title: Max Messages Per Poll
+ description: The most events to receive in one request to SQS, from 1
to 10. SQS may return fewer. When it is not set, it is 10.
+ type: integer
+ minimum: "1"
+ maximum: "10"
+ example: 5
+ waitTimeSeconds:
+ title: Wait Time
+ description: How long one request waits for an event to arrive when
the queue is empty, in seconds, from 0 to 20 (long polling). It saves requests,
and so cost, on a queue that is often empty. When it is not set, the wait time
of the queue is used.
+ example: 20
+ type: integer
+ minimum: "0"
+ maximum: "20"
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long a received event stays hidden from other
readers, in seconds. If it is not deleted by then, it is received again. Set it
longer than processing one event takes, such as fetching a large file with
`getObject`. When it is not set, the visibility timeout of the queue is used
(30 seconds for a new queue).
+ example: 120
+ type: integer
+ minimum: "0"
getObject:
title: Get Object
description: If true, for an event of a file uploaded with a simple
upload (PutObject), it fetches the file and sends its content instead of the
event. It connects to S3 with the same credentials as the queue and with
`region`, not with `uriEndpointOverride`.
@@ -244,6 +264,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
+ waitTimeSeconds: "{{?waitTimeSeconds}}"
+ visibilityTimeout: "{{?visibilityTimeout}}"
steps:
- choice:
precondition: true
diff --git a/kamelets/aws-s3-sink.kamelet.yaml
b/kamelets/aws-s3-sink.kamelet.yaml
index 4a108b14d..ea010fedc 100644
--- a/kamelets/aws-s3-sink.kamelet.yaml
+++ b/kamelets/aws-s3-sink.kamelet.yaml
@@ -41,7 +41,7 @@ spec:
Output: the message is unchanged. The `CamelAwsS3Key` and
`CamelAwsS3ETag` headers give the key and the ETag of the uploaded file.
- Good to know: a file with the same key is replaced without warning. When
`keyName` is set, the `file` header is ignored and every message replaces the
same file. To copy files from `aws-s3-source` under their own names, set the
`file` header from its `CamelAwsS3Key` header first. It also works with
S3-compatible storage, such as MinIO, through `overrideEndpoint`,
`uriEndpointOverride` and `forcePathStyle`. To encrypt the files with a key of
AWS KMS, as a bucket policy may require, [...]
+ Good to know: a file with the same key is replaced without warning,
unless `conditionalWritesEnabled` is true: then the upload of a key that exists
already fails. When `keyName` is set, the `file` header is ignored and every
message replaces the same file. To copy files from `aws-s3-source` under their
own names, set the `file` header from its `CamelAwsS3Key` header first. It also
works with S3-compatible storage, such as MinIO, through `overrideEndpoint`,
`uriEndpointOverride` and [...]
required:
- bucketNameOrArn
- region
@@ -161,6 +161,19 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ multiPartUpload:
+ title: Multipart Upload
+ description: If true, a file of `partSize` bytes or more is uploaded
in parts (an S3 multipart upload), one request per part, and S3 joins the parts
into one file. Smaller files are uploaded in one request, as without it. Use it
for files over 5 GB, which S3 does not take in one request. If the upload
fails, it is aborted, so that S3 discards the parts sent so far, and the
message fails. When it is not set, every file is uploaded in one request.
+ type: boolean
+ partSize:
+ title: Part Size
+ description: "The size of each part of a multipart upload, in bytes,
such as `104857600` for 100 MB; the last part takes the rest. It is also the
size from which a file is uploaded in parts. S3 takes parts of at least 5 MB
(`5242880`), except the last one: with smaller parts, all the parts are sent
and the upload then fails with `EntityTooSmall`. S3 takes at most 10,000 parts
per file. Only used when `multiPartUpload` is true. When it is not set, it is
25 MB (`26214400`)."
+ example: 104857600
+ type: integer
+ conditionalWritesEnabled:
+ title: Conditional Writes
+ description: If true, a file is uploaded only when its key does not
exist yet in the bucket (`If-None-Match`), so an existing file is never
replaced. The upload of a key that exists fails with `PreconditionFailed`
(status 412), and the message fails. Together with `keyName`, only the first
message is stored. When it is not set, an existing file is replaced.
+ type: boolean
dataTypes:
in:
headers:
@@ -214,4 +227,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"aws2-s3:{{bucketNameOrArn}}?accessKey=RAW({{?accessKey}})&autoCreateBucket={{autoCreateBucket}}&forcePathStyle={{forcePathStyle}}&keyName={{?keyName}}&overrideEndpoint={{overrideEndpoint}}&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&uriEndpointOverride={{?uriEndpointOverride}}&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfil
[...]
+ uri:
"aws2-s3:{{bucketNameOrArn}}?accessKey=RAW({{?accessKey}})&autoCreateBucket={{autoCreateBucket}}&forcePathStyle={{forcePathStyle}}&keyName={{?keyName}}&overrideEndpoint={{overrideEndpoint}}&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&uriEndpointOverride={{?uriEndpointOverride}}&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfil
[...]
diff --git a/kamelets/aws-ses-sink.kamelet.yaml
b/kamelets/aws-ses-sink.kamelet.yaml
index c589f9e99..432cd5a13 100644
--- a/kamelets/aws-ses-sink.kamelet.yaml
+++ b/kamelets/aws-ses-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "AWS SES Sink"
description: |-
- Sends each message as a plain-text email through Amazon SES (Simple
Email Service), to the addresses in the `to` header.
+ Sends each message as a plain-text email through Amazon SES (Simple
Email Service), to the addresses in the `to` header or the `to` option.
Use it to send notifications, alerts or confirmations by email from an
integration, such as an order confirmation or an alert when a job fails.
- Input: the body is the text of the email, as plain text. The `to` header
(or `ce-to`) is required: one or more recipient addresses, separated by commas.
The optional `subject` header (or `ce-subject`) is the subject; without it, the
subject is a generated ID. The optional `reply-to-addresses` header (or
`ce-reply-to-addresses`) is a comma-separated list of addresses that replies go
to.
+ Input: the body is the text of the email, as plain text. The `to` header
(or `ce-to`) gives one or more recipient addresses, separated by commas; it is
required unless the `to` option is set. The optional `subject` header (or
`ce-subject`) is the subject; without it and without the `subject` option, the
subject is a generated ID. The optional `reply-to-addresses` header (or
`ce-reply-to-addresses`) is a comma-separated list of addresses that replies go
to. Each of these headers tak [...]
Output: the message is unchanged. The `CamelAwsSesMessageId` header
gives the ID that SES gave the email.
- Good to know: a message without a `to` header fails. The `from` address
must be verified in SES, and while the SES account is in the sandbox, the
recipients must be verified too. To reach AWS through an HTTP proxy, set
`proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP` for a proxy that
takes plain HTTP, such as Squid.
+ Good to know: a message without a `to` header fails, unless the `to`
option is set. The `cc`, `bcc`, `returnPath` and `configurationSet` options
apply to every email. The `from` address, and `returnPath` when it is set, must
be verified in SES, and while the SES account is in the sandbox, the recipients
must be verified too. A `configurationSet` that does not exist in SES makes
every email fail. To reach a local stand-in of AWS, use `overrideEndpoint` and
`uriEndpointOverride`. To [...]
required:
- from
- region
@@ -115,20 +115,70 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ to:
+ title: To
+ description: The addresses to send every email to, separated by
commas, such as `[email protected],[email protected]`. A `to` header on the
message takes precedence. When it is not set, the `to` header is required.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ cc:
+ title: Cc
+ description: The addresses to send a copy of every email to (Cc),
separated by commas. The recipients see them.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ bcc:
+ title: Bcc
+ description: The addresses to send a blind copy of every email to
(Bcc), separated by commas. The other recipients do not see them.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ subject:
+ title: Subject
+ description: The subject of every email. A `subject` header on the
message takes precedence. When it is not set, and there is no `subject` header,
the subject is a generated ID.
+ example: Order confirmation
+ type: string
+ replyToAddresses:
+ title: Reply-To Addresses
+ description: The addresses that replies go to, separated by commas,
such as `[email protected]`. A `reply-to-addresses` header on the message
takes precedence. When it is not set, the emails have no Reply-To, and replies
go to `from`.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ returnPath:
+ title: Return Path
+ description: The address that bounces and complaints are sent to, such
as `[email protected]`. It must be verified in SES, as `from` is. When it is
not set, they go to the `from` address.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+$"
+ configurationSet:
+ title: Configuration Set
+ description: The name of an SES configuration set to send the emails
with, such as `my-configuration-set`, for example to publish their delivery,
bounce, open and click events to CloudWatch, SNS or Firehose. It must exist in
SES in the region, or every email fails with `ConfigurationSetDoesNotExist`.
+ example: my-configuration-set
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ overrideEndpoint:
+ title: Endpoint Overwrite
+ description: If true, it connects to the address in
`uriEndpointOverride` instead of to SES. Use it for a local stand-in of AWS
such as LocalStack or Floci, or for a VPC endpoint of SES.
+ type: boolean
+ uriEndpointOverride:
+ title: Overwrite Endpoint URI
+ description: The address to connect to instead of AWS, as a URL such
as `http://localhost:4566` for LocalStack. Only used when `overrideEndpoint` is
true.
+ example: http://localhost:4566
+ type: string
dataTypes:
in:
headers:
to:
title: To
- description: The addresses to send the email to, separated by
commas. `ce-to` works too. Required.
+ description: The addresses to send the email to, separated by
commas. `ce-to` works too. Required unless the `to` option is set, which it
takes precedence over.
type: string
subject:
title: Subject
- description: The subject of the email. `ce-subject` works too.
Without it, the subject is a generated ID.
+ description: The subject of the email. `ce-subject` works too. It
takes precedence over the `subject` option. Without both, the subject is a
generated ID.
type: string
reply-to-addresses:
title: Reply-To Addresses
- description: The addresses that replies go to, separated by commas.
`ce-reply-to-addresses` works too.
+ description: The addresses that replies go to, separated by commas.
`ce-reply-to-addresses` works too. It takes precedence over the
`replyToAddresses` option.
type: string
dependencies:
- "camel:aws2-ses"
@@ -159,11 +209,18 @@ spec:
expression: "${header[ce-subject]}"
otherwise:
steps:
- - setHeader:
- name: CamelAwsSesSubject
- expression:
- simple:
- expression: "${exchangeId}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:!subject}"
+ steps:
+ - setHeader:
+ name: CamelAwsSesSubject
+ expression:
+ simple:
+ expression: "${exchangeId}"
- choice:
when:
- expression:
@@ -205,4 +262,4 @@ spec:
simple:
expression: "${header[ce-reply-to-addresses]}"
- to:
- uri:
"aws2-ses://{{from}}?accessKey=RAW({{?accessKey}})&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfileCredentialsProvider}}&useSessionCredentials={{useSessionCredentials}}&proxyHost={{?proxyHost}}&proxyPort={{?proxyPort}}&proxyProtocol={{?proxyProtocol}}"
+ uri:
"aws2-ses://{{from}}?accessKey=RAW({{?accessKey}})&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfileCredentialsProvider}}&useSessionCredentials={{useSessionCredentials}}&proxyHost={{?proxyHost}}&proxyPort={{?proxyPort}}&proxyProtocol={{?proxyProtocol}}&to=RAW({{?to}})&cc=RAW({{?cc}}
[...]
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index a653ac9e9..0e3c9aa67 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -195,7 +195,7 @@
"supportLevel" : "Preview",
"title" : "AWS S3 Event Based Source",
"summary" : "Receives the events of an Amazon S3 bucket, such as a new or
a deleted file, from the Amazon SQS queue that Amazon EventBridge sends them
to, one message per event.",
- "description" : "Receives the events of an Amazon S3 bucket, such as a new
or a deleted file, from the Amazon SQS queue that Amazon EventBridge sends them
to, one message per event.\n\nUse it to react to new files in a bucket as soon
as they arrive, without listing the bucket again and again, for example to
process uploaded documents or imported data files. To read the files of a
bucket by polling it, use `aws-s3-source`.\n\nOutput: the body is the
EventBridge event, as JSON text: `d [...]
+ "description" : "Receives the events of an Amazon S3 bucket, such as a new
or a deleted file, from the Amazon SQS queue that Amazon EventBridge sends them
to, one message per event.\n\nUse it to react to new files in a bucket as soon
as they arrive, without listing the bucket again and again, for example to
process uploaded documents or imported data files. To read the files of a
bucket by polling it, use `aws-s3-source`.\n\nOutput: the body is the
EventBridge event, as JSON text: `d [...]
"group" : "AWS S3 Event-Based",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -205,7 +205,7 @@
"supportLevel" : "Stable",
"title" : "AWS S3 Sink",
"summary" : "Uploads each message as a file (object) to an Amazon S3
bucket, under a key (file name) taken from a header, the `keyName` option, or a
generated ID.",
- "description" : "Uploads each message as a file (object) to an Amazon S3
bucket, under a key (file name) taken from a header, the `keyName` option, or a
generated ID.\n\nUse it to store data in S3 for other systems or for archiving,
such as exported records, received documents or events. To collect many small
messages into a few large files, use `aws-s3-streaming-upload-sink`.\n\nInput:
the body is the content of the file, as text, bytes, a stream or a file. Turn
structured data such [...]
+ "description" : "Uploads each message as a file (object) to an Amazon S3
bucket, under a key (file name) taken from a header, the `keyName` option, or a
generated ID.\n\nUse it to store data in S3 for other systems or for archiving,
such as exported records, received documents or events. To collect many small
messages into a few large files, use `aws-s3-streaming-upload-sink`.\n\nInput:
the body is the content of the file, as text, bytes, a stream or a file. Turn
structured data such [...]
"group" : "AWS S3",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -244,8 +244,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "AWS SES Sink",
- "summary" : "Sends each message as a plain-text email through Amazon SES
(Simple Email Service), to the addresses in the `to` header.",
- "description" : "Sends each message as a plain-text email through Amazon
SES (Simple Email Service), to the addresses in the `to` header.\n\nUse it to
send notifications, alerts or confirmations by email from an integration, such
as an order confirmation or an alert when a job fails.\n\nInput: the body is
the text of the email, as plain text. The `to` header (or `ce-to`) is required:
one or more recipient addresses, separated by commas. The optional `subject`
header (or `ce-subject`) [...]
+ "summary" : "Sends each message as a plain-text email through Amazon SES
(Simple Email Service), to the addresses in the `to` header or the `to`
option.",
+ "description" : "Sends each message as a plain-text email through Amazon
SES (Simple Email Service), to the addresses in the `to` header or the `to`
option.\n\nUse it to send notifications, alerts or confirmations by email from
an integration, such as an order confirmation or an alert when a job
fails.\n\nInput: the body is the text of the email, as plain text. The `to`
header (or `ce-to`) gives one or more recipient addresses, separated by commas;
it is required unless the `to` opti [...]
"group" : "AWS SES",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-s3-event-based-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-s3-event-based-source.kamelet.yaml
index a26cfa578..4db8b9cdb 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-s3-event-based-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-s3-event-based-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the EventBridge event, as JSON text: `detail-type`
says what happened (such as `Object Created` or `Object Deleted`),
`detail.bucket.name` is the bucket and `detail.object.key` the key of the file.
With `getObject` set to true, the event of a file uploaded with a simple upload
(PutObject) carries the content of the file instead, as a stream, with the
`CamelAwsS3ProducedBucketName` and `CamelAwsS3ProducedKey` headers; the other
events then come as parsed JSON (a [...]
- Good to know: it needs a setup in AWS. In the properties of the bucket,
turn on sending events to Amazon EventBridge; then create an EventBridge rule
for the events of source `aws.s3` and that bucket, with the queue as its
target. Each event is deleted from the queue once it has been processed, unless
`deleteAfterRead` is false. The files stay in the bucket. `getObject` skips
files created by a multipart upload or a copy, and never connects through
`uriEndpointOverride`. It logs in [...]
+ Good to know: it needs a setup in AWS. In the properties of the bucket,
turn on sending events to Amazon EventBridge; then create an EventBridge rule
for the events of source `aws.s3` and that bucket, with the queue as its
target. Each event is deleted from the queue once it has been processed, unless
`deleteAfterRead` is false; an event that fails stays in the queue and is
received again when its visibility timeout (`visibilityTimeout`) ends. The
files stay in the bucket. `getObje [...]
required:
- queueNameOrArn
- region
@@ -176,6 +176,26 @@ spec:
description: If true, it polls again right away when the last poll
received messages, instead of waiting for `delay`.
type: boolean
default: false
+ maxMessagesPerPoll:
+ title: Max Messages Per Poll
+ description: The most events to receive in one request to SQS, from 1
to 10. SQS may return fewer. When it is not set, it is 10.
+ type: integer
+ minimum: "1"
+ maximum: "10"
+ example: 5
+ waitTimeSeconds:
+ title: Wait Time
+ description: How long one request waits for an event to arrive when
the queue is empty, in seconds, from 0 to 20 (long polling). It saves requests,
and so cost, on a queue that is often empty. When it is not set, the wait time
of the queue is used.
+ example: 20
+ type: integer
+ minimum: "0"
+ maximum: "20"
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long a received event stays hidden from other
readers, in seconds. If it is not deleted by then, it is received again. Set it
longer than processing one event takes, such as fetching a large file with
`getObject`. When it is not set, the visibility timeout of the queue is used
(30 seconds for a new queue).
+ example: 120
+ type: integer
+ minimum: "0"
getObject:
title: Get Object
description: If true, for an event of a file uploaded with a simple
upload (PutObject), it fetches the file and sends its content instead of the
event. It connects to S3 with the same credentials as the queue and with
`region`, not with `uriEndpointOverride`.
@@ -244,6 +264,9 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
+ waitTimeSeconds: "{{?waitTimeSeconds}}"
+ visibilityTimeout: "{{?visibilityTimeout}}"
steps:
- choice:
precondition: true
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-s3-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-s3-sink.kamelet.yaml
index 4a108b14d..ea010fedc 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-s3-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-s3-sink.kamelet.yaml
@@ -41,7 +41,7 @@ spec:
Output: the message is unchanged. The `CamelAwsS3Key` and
`CamelAwsS3ETag` headers give the key and the ETag of the uploaded file.
- Good to know: a file with the same key is replaced without warning. When
`keyName` is set, the `file` header is ignored and every message replaces the
same file. To copy files from `aws-s3-source` under their own names, set the
`file` header from its `CamelAwsS3Key` header first. It also works with
S3-compatible storage, such as MinIO, through `overrideEndpoint`,
`uriEndpointOverride` and `forcePathStyle`. To encrypt the files with a key of
AWS KMS, as a bucket policy may require, [...]
+ Good to know: a file with the same key is replaced without warning,
unless `conditionalWritesEnabled` is true: then the upload of a key that exists
already fails. When `keyName` is set, the `file` header is ignored and every
message replaces the same file. To copy files from `aws-s3-source` under their
own names, set the `file` header from its `CamelAwsS3Key` header first. It also
works with S3-compatible storage, such as MinIO, through `overrideEndpoint`,
`uriEndpointOverride` and [...]
required:
- bucketNameOrArn
- region
@@ -161,6 +161,19 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ multiPartUpload:
+ title: Multipart Upload
+ description: If true, a file of `partSize` bytes or more is uploaded
in parts (an S3 multipart upload), one request per part, and S3 joins the parts
into one file. Smaller files are uploaded in one request, as without it. Use it
for files over 5 GB, which S3 does not take in one request. If the upload
fails, it is aborted, so that S3 discards the parts sent so far, and the
message fails. When it is not set, every file is uploaded in one request.
+ type: boolean
+ partSize:
+ title: Part Size
+ description: "The size of each part of a multipart upload, in bytes,
such as `104857600` for 100 MB; the last part takes the rest. It is also the
size from which a file is uploaded in parts. S3 takes parts of at least 5 MB
(`5242880`), except the last one: with smaller parts, all the parts are sent
and the upload then fails with `EntityTooSmall`. S3 takes at most 10,000 parts
per file. Only used when `multiPartUpload` is true. When it is not set, it is
25 MB (`26214400`)."
+ example: 104857600
+ type: integer
+ conditionalWritesEnabled:
+ title: Conditional Writes
+ description: If true, a file is uploaded only when its key does not
exist yet in the bucket (`If-None-Match`), so an existing file is never
replaced. The upload of a key that exists fails with `PreconditionFailed`
(status 412), and the message fails. Together with `keyName`, only the first
message is stored. When it is not set, an existing file is replaced.
+ type: boolean
dataTypes:
in:
headers:
@@ -214,4 +227,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"aws2-s3:{{bucketNameOrArn}}?accessKey=RAW({{?accessKey}})&autoCreateBucket={{autoCreateBucket}}&forcePathStyle={{forcePathStyle}}&keyName={{?keyName}}&overrideEndpoint={{overrideEndpoint}}&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&uriEndpointOverride={{?uriEndpointOverride}}&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfil
[...]
+ uri:
"aws2-s3:{{bucketNameOrArn}}?accessKey=RAW({{?accessKey}})&autoCreateBucket={{autoCreateBucket}}&forcePathStyle={{forcePathStyle}}&keyName={{?keyName}}&overrideEndpoint={{overrideEndpoint}}&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&uriEndpointOverride={{?uriEndpointOverride}}&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfil
[...]
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-ses-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-ses-sink.kamelet.yaml
index c589f9e99..432cd5a13 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-ses-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-ses-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "AWS SES Sink"
description: |-
- Sends each message as a plain-text email through Amazon SES (Simple
Email Service), to the addresses in the `to` header.
+ Sends each message as a plain-text email through Amazon SES (Simple
Email Service), to the addresses in the `to` header or the `to` option.
Use it to send notifications, alerts or confirmations by email from an
integration, such as an order confirmation or an alert when a job fails.
- Input: the body is the text of the email, as plain text. The `to` header
(or `ce-to`) is required: one or more recipient addresses, separated by commas.
The optional `subject` header (or `ce-subject`) is the subject; without it, the
subject is a generated ID. The optional `reply-to-addresses` header (or
`ce-reply-to-addresses`) is a comma-separated list of addresses that replies go
to.
+ Input: the body is the text of the email, as plain text. The `to` header
(or `ce-to`) gives one or more recipient addresses, separated by commas; it is
required unless the `to` option is set. The optional `subject` header (or
`ce-subject`) is the subject; without it and without the `subject` option, the
subject is a generated ID. The optional `reply-to-addresses` header (or
`ce-reply-to-addresses`) is a comma-separated list of addresses that replies go
to. Each of these headers tak [...]
Output: the message is unchanged. The `CamelAwsSesMessageId` header
gives the ID that SES gave the email.
- Good to know: a message without a `to` header fails. The `from` address
must be verified in SES, and while the SES account is in the sandbox, the
recipients must be verified too. To reach AWS through an HTTP proxy, set
`proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP` for a proxy that
takes plain HTTP, such as Squid.
+ Good to know: a message without a `to` header fails, unless the `to`
option is set. The `cc`, `bcc`, `returnPath` and `configurationSet` options
apply to every email. The `from` address, and `returnPath` when it is set, must
be verified in SES, and while the SES account is in the sandbox, the recipients
must be verified too. A `configurationSet` that does not exist in SES makes
every email fail. To reach a local stand-in of AWS, use `overrideEndpoint` and
`uriEndpointOverride`. To [...]
required:
- from
- region
@@ -115,20 +115,70 @@ spec:
type: string
enum: ["HTTP", "HTTPS"]
example: HTTP
+ to:
+ title: To
+ description: The addresses to send every email to, separated by
commas, such as `[email protected],[email protected]`. A `to` header on the
message takes precedence. When it is not set, the `to` header is required.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ cc:
+ title: Cc
+ description: The addresses to send a copy of every email to (Cc),
separated by commas. The recipients see them.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ bcc:
+ title: Bcc
+ description: The addresses to send a blind copy of every email to
(Bcc), separated by commas. The other recipients do not see them.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ subject:
+ title: Subject
+ description: The subject of every email. A `subject` header on the
message takes precedence. When it is not set, and there is no `subject` header,
the subject is a generated ID.
+ example: Order confirmation
+ type: string
+ replyToAddresses:
+ title: Reply-To Addresses
+ description: The addresses that replies go to, separated by commas,
such as `[email protected]`. A `reply-to-addresses` header on the message
takes precedence. When it is not set, the emails have no Reply-To, and replies
go to `from`.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$"
+ returnPath:
+ title: Return Path
+ description: The address that bounces and complaints are sent to, such
as `[email protected]`. It must be verified in SES, as `from` is. When it is
not set, they go to the `from` address.
+ example: [email protected]
+ type: string
+ pattern: "^[^ ,@]+@[^ ,@]+$"
+ configurationSet:
+ title: Configuration Set
+ description: The name of an SES configuration set to send the emails
with, such as `my-configuration-set`, for example to publish their delivery,
bounce, open and click events to CloudWatch, SNS or Firehose. It must exist in
SES in the region, or every email fails with `ConfigurationSetDoesNotExist`.
+ example: my-configuration-set
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ overrideEndpoint:
+ title: Endpoint Overwrite
+ description: If true, it connects to the address in
`uriEndpointOverride` instead of to SES. Use it for a local stand-in of AWS
such as LocalStack or Floci, or for a VPC endpoint of SES.
+ type: boolean
+ uriEndpointOverride:
+ title: Overwrite Endpoint URI
+ description: The address to connect to instead of AWS, as a URL such
as `http://localhost:4566` for LocalStack. Only used when `overrideEndpoint` is
true.
+ example: http://localhost:4566
+ type: string
dataTypes:
in:
headers:
to:
title: To
- description: The addresses to send the email to, separated by
commas. `ce-to` works too. Required.
+ description: The addresses to send the email to, separated by
commas. `ce-to` works too. Required unless the `to` option is set, which it
takes precedence over.
type: string
subject:
title: Subject
- description: The subject of the email. `ce-subject` works too.
Without it, the subject is a generated ID.
+ description: The subject of the email. `ce-subject` works too. It
takes precedence over the `subject` option. Without both, the subject is a
generated ID.
type: string
reply-to-addresses:
title: Reply-To Addresses
- description: The addresses that replies go to, separated by commas.
`ce-reply-to-addresses` works too.
+ description: The addresses that replies go to, separated by commas.
`ce-reply-to-addresses` works too. It takes precedence over the
`replyToAddresses` option.
type: string
dependencies:
- "camel:aws2-ses"
@@ -159,11 +209,18 @@ spec:
expression: "${header[ce-subject]}"
otherwise:
steps:
- - setHeader:
- name: CamelAwsSesSubject
- expression:
- simple:
- expression: "${exchangeId}"
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${propertiesExist:!subject}"
+ steps:
+ - setHeader:
+ name: CamelAwsSesSubject
+ expression:
+ simple:
+ expression: "${exchangeId}"
- choice:
when:
- expression:
@@ -205,4 +262,4 @@ spec:
simple:
expression: "${header[ce-reply-to-addresses]}"
- to:
- uri:
"aws2-ses://{{from}}?accessKey=RAW({{?accessKey}})&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfileCredentialsProvider}}&useSessionCredentials={{useSessionCredentials}}&proxyHost={{?proxyHost}}&proxyPort={{?proxyPort}}&proxyProtocol={{?proxyProtocol}}"
+ uri:
"aws2-ses://{{from}}?accessKey=RAW({{?accessKey}})&profileCredentialsName={{?profileCredentialsName}}®ion={{region}}&secretKey=RAW({{?secretKey}})&sessionToken=RAW({{?sessionToken}})&useDefaultCredentialsProvider={{useDefaultCredentialsProvider}}&useProfileCredentialsProvider={{useProfileCredentialsProvider}}&useSessionCredentials={{useSessionCredentials}}&proxyHost={{?proxyHost}}&proxyPort={{?proxyPort}}&proxyProtocol={{?proxyProtocol}}&to=RAW({{?to}})&cc=RAW({{?cc}}
[...]