This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new a1b06e237 chore: the Kinesis and DynamoDB Streams sources can limit a
poll, the CloudWatch sink can take the metric as options, the CloudTrail source
can look back for late events (#3170) (#3233)
a1b06e237 is described below
commit a1b06e237297f5cdc216ec532e76326c07abbb20
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:37:47 2026 +0100
chore: the Kinesis and DynamoDB Streams sources can limit a poll, the
CloudWatch sink can take the metric as options, the CloudTrail source can look
back for late events (#3170) (#3233)
* chore: the Kinesis and DynamoDB Streams sources can limit a poll, the
CloudWatch sink can take the metric as options, the CloudTrail source can look
back for late events (#3170)
New optional options, mapped with optional placeholders, so with them
unset the endpoint URIs are the same as before:
- aws-kinesis-source: maxResultsPerRequest (records per shard and poll),
shardId (read one shard), shardClosed (ignore, silent or fail when a
shard is closed by resharding) and applicationName (the KCL
application, which names its DynamoDB lease table).
- aws-cloudwatch-sink: metricName, metricValue and metricUnit, mapped to
the name, value and unit of the endpoint, for the messages without the
metric-name, metric-value and metric-unit headers. metricName goes into
RAW() so that a + or % in the name is kept.
- aws-cloudtrail-source: lookBackWindow and lookBackInterval (Camel 4.24),
to read the events that CloudTrail makes visible after a newer one.
- aws-ddb-streams-source: maxResultsPerRequest (changes per poll).
The aws-cloudtrail-source description said it also reads the events from
before it starts, and reads them again after a restart. Since Camel 4.22
it starts at the time it starts, so the description now says so.
aws-ddb-sink does not get consistentRead: the option only applies to
GetItem, Query, Scan and ExecuteStatement, and the JSON transformation of
the Kamelet supports PutItem, UpdateItem and DeleteItem only.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* chore: aws-kinesis-source with shardClosed=fail still reads the other
shards (#3170)
The test log shows that the records of the new shards are still read;
each poll fails and logs an error.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../aws-cloudtrail-source-description.adoc | 19 +++++++++++++
.../partials/aws-cloudwatch-sink-description.adoc | 17 ++++++++++++
.../aws-ddb-streams-source-description.adoc | 20 ++++++++++++++
.../partials/aws-kinesis-source-description.adoc | 25 +++++++++++++++++
kamelets/aws-cloudtrail-source.kamelet.yaml | 14 +++++++++-
kamelets/aws-cloudwatch-sink.kamelet.yaml | 32 ++++++++++++++++++----
kamelets/aws-ddb-streams-source.kamelet.yaml | 8 +++++-
kamelets/aws-kinesis-source.kamelet.yaml | 29 +++++++++++++++++++-
.../main/resources/META-INF/kamelets-index.json | 10 +++----
.../kamelets/aws-cloudtrail-source.kamelet.yaml | 14 +++++++++-
.../kamelets/aws-cloudwatch-sink.kamelet.yaml | 32 ++++++++++++++++++----
.../kamelets/aws-ddb-streams-source.kamelet.yaml | 8 +++++-
.../kamelets/aws-kinesis-source.kamelet.yaml | 29 +++++++++++++++++++-
13 files changed, 234 insertions(+), 23 deletions(-)
diff --git a/docs/modules/ROOT/partials/aws-cloudtrail-source-description.adoc
b/docs/modules/ROOT/partials/aws-cloudtrail-source-description.adoc
index eba208365..4c3b4de67 100644
--- a/docs/modules/ROOT/partials/aws-cloudtrail-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-cloudtrail-source-description.adoc
@@ -19,6 +19,25 @@ Only one of access key/secret key or default credentials
provider could be used
For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html[AWS
credentials documentation]
+=== Events That Become Visible Late
+
+The Kamelet reads the events that happen after it starts. CloudTrail can make
an event visible some minutes after it happened, and after a newer event, which
the Kamelet has then already read past. So every `lookBackInterval` (one minute
when it is not set) the Kamelet looks up the last `lookBackWindow` (ten minutes
when it is not set) again, and reads the events it has not read yet. Set
`lookBackWindow` to 0 to turn this off, or set a shorter `lookBackInterval` to
read a late event soon [...]
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:aws-cloudtrail-source"
+ parameters:
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ lookBackWindow: 900000
+ lookBackInterval: 30000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-cloudwatch-sink-description.adoc
b/docs/modules/ROOT/partials/aws-cloudwatch-sink-description.adoc
index 7b54219d7..b679580e1 100644
--- a/docs/modules/ROOT/partials/aws-cloudwatch-sink-description.adoc
+++ b/docs/modules/ROOT/partials/aws-cloudwatch-sink-description.adoc
@@ -30,6 +30,23 @@ In this Kamelet you could use the following headers for
metrics:
- `metric-dimension-name` / `ce-metricdimensionname` for the dimension name.
- `metric-dimension-value` / `ce-metricdimensionvalue` for the dimension value.
+=== The Same Metric for Every Message
+
+To send the same metric for every message, set `metricName`, and `metricValue`
and `metricUnit` if needed, instead of the headers. A header wins over the
option, so a message can still send another metric. Without a value, the value
is 1, and without a unit, the unit is `Count`: with only `metricName`, the
metric counts the messages.
+
+The step that sends to the Kamelet, in a route:
+
+[source,yaml,subs='+attributes,macros']
+----
+- to:
+ uri: "kamelet:aws-cloudwatch-sink"
+ parameters:
+ cwNamespace: "MyApp/Orders"
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ metricName: "OrdersProcessed"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-ddb-streams-source-description.adoc
b/docs/modules/ROOT/partials/aws-ddb-streams-source-description.adoc
index 4fdb0a062..566fb706f 100644
--- a/docs/modules/ROOT/partials/aws-ddb-streams-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-ddb-streams-source-description.adoc
@@ -19,6 +19,26 @@ Only one of access key/secret key or default credentials
provider could be used
For more information, see the
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html[AWS
credentials documentation]
+=== Fewer Changes in One Poll
+
+The Kamelet reads up to 100 changes in one poll, and polls again `delay`
milliseconds later. To read fewer at a time, for example so that a slow route
behind it is not handed a large burst at once, set `maxResultsPerRequest`. The
number is shared between the shards of the stream. Here a burst of 1000 changes
is read 50 at a time, one poll a second:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:aws-ddb-streams-source"
+ parameters:
+ table: "orders"
+ region: "eu-west-1"
+ useDefaultCredentialsProvider: true
+ maxResultsPerRequest: 50
+ delay: 1000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
b/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
index 3e3941220..ff5c929b8 100644
--- a/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
@@ -58,6 +58,28 @@ By default (`iteratorType` is `TRIM_HORIZON`) the Kamelet
reads the stream from
The records written while the integration was stopped are then not read. To
start at a given record, set `iteratorType` to `AT_SEQUENCE_NUMBER` or
`AFTER_SEQUENCE_NUMBER` and `sequenceNumber` to its sequence number; to start
at a given time, set `iteratorType` to `AT_TIMESTAMP` and `messageTimestamp` to
the time, such as `2026-11-17T08:00:00Z`. The Kamelet does not remember where
it stopped: it starts at the same place on each start. The KCL consumer below
does not use `iteratorType`.
+=== One shard, and smaller polls
+
+By default the Kamelet reads all the shards of the stream, and up to 1000
records from each shard in one poll. To read one shard only, set `shardId` to
its ID, as `aws kinesis list-shards` shows it. To read fewer records in one
poll, set `maxResultsPerRequest`: the rest of the records are read in the next
polls, `delay` apart.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:aws-kinesis-source"
+ parameters:
+ useDefaultCredentialsProvider: true
+ region: "eu-west-1"
+ stream: "kamelets"
+ shardId: "shardId-000000000001"
+ maxResultsPerRequest: 100
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+When the stream is resharded, for example when the number of shards is
changed, the old shards are closed and new ones take their place. Without
`shardId`, the Kamelet goes on with the new shards, and `shardClosed` says what
it does about a closed shard once it has read its last record: `ignore` (the
default) logs a warning once, and `silent` logs nothing. With `fail`, each
later poll fails with an error and a stack trace in the log, for as long as the
closed shard is listed, which is 24 [...]
+
=== Usage example with KCL Consumer
You could consume the stream content with the KCL support
@@ -73,6 +95,7 @@ You could consume the stream content with the KCL support
region: "eu-west-1"
asyncClient: true
useKclConsumers: true
+ applicationName: "orders-processor"
steps:
- to:
uri: "kamelet:log-sink"
@@ -84,6 +107,8 @@ With the `useKclConsumers` enabled, you won't have to deal
with shard iteration
As a side note you need to remember that the KCL consumer will need access to
DynamoDB and Cloudwatch services from AWS, so it will create clients to these
services under the hood and it will use them.
+The KCL keeps its state in a DynamoDB table named after `applicationName`,
`orders-processor` above, which it creates; without `applicationName`, the
table is named after the stream. Copies of the integration with the same
`applicationName` share the shards of the stream. Give each integration that
reads the same stream its own `applicationName`, so that each one reads all the
records.
+
=== Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company
networks, set `proxyHost` and `proxyPort` to the proxy. `proxyProtocol` is how
the Kamelet talks to the proxy itself, not to AWS: it is `HTTPS` when it is not
set, so set it to `HTTP` for a proxy that takes plain HTTP, such as Squid. The
calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has
no option for a proxy user or password.
diff --git a/kamelets/aws-cloudtrail-source.kamelet.yaml
b/kamelets/aws-cloudtrail-source.kamelet.yaml
index 6f25ad7e6..740531e97 100644
--- a/kamelets/aws-cloudtrail-source.kamelet.yaml
+++ b/kamelets/aws-cloudtrail-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the CloudTrail record of the event, as JSON text.
The headers give the ID, the name (such as `GetSecretValue`), the source (such
as `secretsmanager.amazonaws.com`) and the user name of the event.
- Good to know: it looks up the management events that CloudTrail keeps
for the region, which go back 90 days, so when it starts it also reads events
from before. It remembers which events it read only while it runs: after a
restart it reads them again. Nothing is removed from CloudTrail. An event can
be looked up only some minutes after it happened. To reach AWS through an HTTP
proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP` for a
proxy that takes plain HTT [...]
+ Good to know: it looks up the management events that CloudTrail keeps
for the region, and reads the events that happen after it starts. It remembers
where it is only while it runs: the events that happen while it is stopped are
not read. Nothing is removed from CloudTrail. An event can be looked up only
some minutes after it happened, and sometimes after a newer event: every
`lookBackInterval` it looks up the last `lookBackWindow` again, and reads the
events it missed. To reach AWS [...]
required:
- region
type: object
@@ -110,6 +110,16 @@ spec:
description: Read only the events of this AWS service, given as its
address, such as `s3.amazonaws.com` or `secretsmanager.amazonaws.com`. When it
is not set, the events of all services are read.
type: string
example: secretsmanager.amazonaws.com
+ lookBackWindow:
+ title: Look Back Window
+ description: How far back it looks up the events again, in
milliseconds, to read the events that CloudTrail made visible after a newer
one. When it is not set, it is 600000, ten minutes. 0 turns the second lookup
off, and then such a late event is not read. Needs Camel 4.24 or later.
+ type: integer
+ example: 600000
+ lookBackInterval:
+ title: Look Back Interval
+ description: How often it looks up the `lookBackWindow` again, in
milliseconds. A late event is read at most this long after it became visible.
When it is not set, it is 60000, one minute. It must be above 0 unless
`lookBackWindow` is 0, or the integration does not start. Needs Camel 4.24 or
later.
+ type: integer
+ example: 60000
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -200,6 +210,8 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ lookBackWindow: "{{?lookBackWindow}}"
+ lookBackInterval: "{{?lookBackInterval}}"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/aws-cloudwatch-sink.kamelet.yaml
b/kamelets/aws-cloudwatch-sink.kamelet.yaml
index 905463ee8..b6da2a46a 100644
--- a/kamelets/aws-cloudwatch-sink.kamelet.yaml
+++ b/kamelets/aws-cloudwatch-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "AWS CloudWatch Metrics Sink"
description: |-
- Sends one data point of a custom metric to Amazon CloudWatch for each
message, with the name, the value and the unit of the metric taken from headers.
+ Sends one data point of a custom metric to Amazon CloudWatch for each
message, with the name, the value and the unit of the metric taken from headers
or options.
Use it to record business or technical numbers from an integration, such
as orders processed, items in stock or processing time, and to graph them or
set alarms on them in CloudWatch.
- Input: the body is not used. The `metric-name` header (or
`ce-metricname`) is the name of the metric, and is required. The optional
`metric-value` header (or `ce-metricvalue`) is the value, a number, 1 when it
is not set. The optional `metric-unit` header (or `ce-metricunit`) is the unit,
such as `Count`, `Seconds`, `Bytes` or `Percent`, `Count` when it is not set.
The optional `metric-dimension-name` and `metric-dimension-value` headers (or
`ce-metricdimensionname` and `ce-metricd [...]
+ Input: the body is not used. The `metric-name` header (or
`ce-metricname`) is the name of the metric; it is required unless the
`metricName` option is set. The optional `metric-value` header (or
`ce-metricvalue`) is the value, a number; when it is not set, the value is
`metricValue`, or 1. The optional `metric-unit` header (or `ce-metricunit`) is
the unit, such as `Count`, `Seconds`, `Bytes` or `Percent`; when it is not set,
the unit is `metricUnit`, or `Count`. A header wins over [...]
Output: the message is unchanged.
- Good to know: a message without a metric name, or with a value that is
not a number, fails. The time in `metric-timestamp` must be a Java `Instant` or
`Date`: text is ignored, and then the data point gets the time it is sent. To
reach AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: a message without a metric name, when `metricName` is not
set either, fails, and so does a value that is not a number or a unit that
CloudWatch does not know. Set the options to send the same metric for every
message, for example `metricName` `OrdersProcessed` to count the messages. The
time in `metric-timestamp` must be a Java `Instant` or `Date`: text is ignored,
and then the data point gets the time it is sent. To reach AWS through an HTTP
proxy, set `proxyHost` an [...]
required:
- cwNamespace
- region
@@ -51,6 +51,23 @@ spec:
description: The namespace that groups your metrics in CloudWatch,
such as `MyApp/Orders`. It must not start with `AWS/`, which is kept for the
metrics of AWS services.
example: MyApp/Orders
type: string
+ metricName:
+ title: Metric Name
+ description: The name of the metric, such as `OrdersProcessed`, for
the messages without a `metric-name` header. Up to 255 characters, which can be
letters, digits, spaces and punctuation. When neither is set, the message fails.
+ type: string
+ pattern: "^[ -~]{1,255}$"
+ example: OrdersProcessed
+ metricValue:
+ title: Metric Value
+ description: The value of the data point, a number, for the messages
without a `metric-value` header. When neither is set, the value is 1.
+ type: number
+ example: 1
+ metricUnit:
+ title: Metric Unit
+ description: The unit of the value, for the messages without a
`metric-unit` header. When neither is set, the unit is `Count`.
+ type: string
+ enum: ["Seconds", "Microseconds", "Milliseconds", "Bytes",
"Kilobytes", "Megabytes", "Gigabytes", "Terabytes", "Bits", "Kilobits",
"Megabits", "Gigabits", "Terabits", "Percent", "Count", "Bytes/Second",
"Kilobytes/Second", "Megabytes/Second", "Gigabytes/Second", "Terabytes/Second",
"Bits/Second", "Kilobits/Second", "Megabits/Second", "Gigabits/Second",
"Terabits/Second", "Count/Second", "None"]
+ example: Milliseconds
accessKey:
title: Access Key
description: The access key ID of an IAM user, which you create in the
AWS console under IAM, Users, Security credentials. Use it together with
`secretKey`. Leave both out when `useDefaultCredentialsProvider` or
`useProfileCredentialsProvider` is true.
@@ -130,15 +147,15 @@ spec:
headers:
metric-name:
title: Metric Name
- description: The name of the metric, such as `OrdersProcessed`.
`ce-metricname` works too. Required.
+ description: The name of the metric, such as `OrdersProcessed`.
`ce-metricname` works too. Required unless the `metricName` option is set.
type: string
metric-value:
title: Metric Value
- description: The value of the data point, a number. `ce-metricvalue`
works too. 1 when it is not set.
+ description: The value of the data point, a number. `ce-metricvalue`
works too. When it is not set, the value is the `metricValue` option, or 1.
type: number
metric-unit:
title: Metric Unit
- description: The unit of the value, such as `Count`, `Seconds`,
`Bytes` or `Percent`. `ce-metricunit` works too. `Count` when it is not set.
+ description: The unit of the value, such as `Count`, `Seconds`,
`Bytes` or `Percent`. `ce-metricunit` works too. When it is not set, the unit
is the `metricUnit` option, or `Count`.
type: string
metric-dimension-name:
title: Dimension Name
@@ -296,3 +313,6 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ name: "RAW({{?metricName}})"
+ value: "{{?metricValue}}"
+ unit: "{{?metricUnit}}"
diff --git a/kamelets/aws-ddb-streams-source.kamelet.yaml
b/kamelets/aws-ddb-streams-source.kamelet.yaml
index 6002fc0d5..56c4190df 100644
--- a/kamelets/aws-ddb-streams-source.kamelet.yaml
+++ b/kamelets/aws-ddb-streams-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is a JSON object, as JSON text, that describes one
change: `eventName` is `INSERT`, `MODIFY` or `REMOVE`, and `dynamodb` holds the
key of the item (`keys`) and, depending on the view type of the stream, the
item before (`oldImage`) and after (`newImage`) the change, with the type of
each attribute. The headers give the ID and the name of the event.
- Good to know: DynamoDB Streams must be enabled on the table, with the
view type that holds the images you need, such as new and old images. By
default it reads only the changes made after it starts. To reach AWS through an
HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP`
for a proxy that takes plain HTTP, such as Squid.
+ Good to know: DynamoDB Streams must be enabled on the table, with the
view type that holds the images you need, such as new and old images. By
default it reads only the changes made after it starts, up to 100 in one poll;
set `maxResultsPerRequest` to read fewer at a time. To reach AWS through an
HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP`
for a proxy that takes plain HTTP, such as Squid.
required:
- table
- region
@@ -116,6 +116,11 @@ spec:
description: How long to wait between two reads of the stream, in
milliseconds.
type: integer
default: 500
+ maxResultsPerRequest:
+ title: Max Changes Per Poll
+ description: The most changes it reads in one poll, from 1 to 1000.
The number is shared between the shards of the stream that it reads, at least 1
each. When it is not set, it is 100. With a lower number, a burst of changes
takes more polls, `delay` apart, to read.
+ type: integer
+ example: 50
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -203,6 +208,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxResultsPerRequest: "{{?maxResultsPerRequest}}"
steps:
- transformDataType:
toType: aws2-ddb:application-x-struct
diff --git a/kamelets/aws-kinesis-source.kamelet.yaml
b/kamelets/aws-kinesis-source.kamelet.yaml
index 775e05694..e48136fca 100644
--- a/kamelets/aws-kinesis-source.kamelet.yaml
+++ b/kamelets/aws-kinesis-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the data of the record, as bytes. The headers give
the partition key, the sequence number, the shard ID and the arrival time of
the record.
- Good to know: by default it reads the stream from its oldest record, and
again from the oldest record each time the integration starts: it does not
remember where it stopped, so records are read again after a restart. Set
`iteratorType` to `LATEST` to read only the records written after it started;
the records written while it was stopped are then not read. `iteratorType` can
also start at a sequence number or at a time, again on each start. Records are
not removed: Kinesis keeps t [...]
+ Good to know: by default it reads the stream from its oldest record, and
again from the oldest record each time the integration starts: it does not
remember where it stopped, so records are read again after a restart. Set
`iteratorType` to `LATEST` to read only the records written after it started;
the records written while it was stopped are then not read. `iteratorType` can
also start at a sequence number or at a time, again on each start. Records are
not removed: Kinesis keeps t [...]
required:
- stream
- region
@@ -128,6 +128,23 @@ spec:
description: How long to wait between two polls of the stream, in
milliseconds.
type: integer
default: 500
+ maxResultsPerRequest:
+ title: Max Records Per Poll
+ description: The most records it reads from each shard in one poll,
from 1 to 10000. When it is not set, it is 1000. Not used when
`useKclConsumers` is true.
+ type: integer
+ example: 100
+ shardId:
+ title: Shard ID
+ description: The ID of the one shard to read, such as
`shardId-000000000000`, as the AWS console or `aws kinesis list-shards` shows
it. When it is not set, it reads all the shards of the stream. Not used when
`useKclConsumers` is true.
+ type: string
+ pattern: "^shardId-[0-9]+$"
+ example: shardId-000000000000
+ shardClosed:
+ title: Closed Shard
+ description: "What to do when it has read the last record of a closed
shard. A shard is closed when the stream is resharded, for example when the
number of shards is changed, and new shards take its place. `ignore` logs a
warning once and goes on with the other shards, and so does `silent`, without
the warning. With `fail`, each later poll fails with an error and a stack trace
in the log, for as long as the closed shard is listed, which is 24 hours by
default; the records of the [...]
+ type: string
+ enum: ["ignore", "silent", "fail"]
+ example: ignore
asyncClient:
title: Async Client
description: If true, it uses the asynchronous Kinesis client of the
AWS SDK. `useKclConsumers` needs it.
@@ -143,6 +160,12 @@ spec:
description: If true, the Kinesis Client Library does not send its
metrics to CloudWatch. Only used when `useKclConsumers` is true.
type: boolean
default: false
+ applicationName:
+ title: KCL Application Name
+ description: "The name of the application for the Kinesis Client
Library. The KCL keeps where it stopped, and which copy of the integration
reads which shard, in a DynamoDB table of this name, which it creates. Copies
of the integration with the same name share the shards of the stream; give
another integration that reads the same stream its own name, so that it reads
all the records too. When it is not set, it is the name of the stream. Only
used when `useKclConsumers` is true."
+ type: string
+ pattern: "^[A-Za-z0-9_.-]{3,255}$"
+ example: orders-processor
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -210,6 +233,10 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxResultsPerRequest: "{{?maxResultsPerRequest}}"
+ shardId: "{{?shardId}}"
+ shardClosed: "{{?shardClosed}}"
+ applicationName: "{{?applicationName}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 1c14d4648..962a83ce4 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -55,7 +55,7 @@
"supportLevel" : "Stable",
"title" : "AWS Cloudtrail Source",
"summary" : "Reads the events that AWS CloudTrail records in an AWS
account, such as API calls and console sign-ins, one message per event.",
- "description" : "Reads the events that AWS CloudTrail records in an AWS
account, such as API calls and console sign-ins, one message per event.\n\nUse
it to react to activity in an AWS account, for example to raise an alert when a
secret is read, a security group is changed or someone signs in to the console.
Set `eventSource` to read the events of one AWS service only.\n\nOutput: the
body is the CloudTrail record of the event, as JSON text. The headers give the
ID, the name (such as [...]
+ "description" : "Reads the events that AWS CloudTrail records in an AWS
account, such as API calls and console sign-ins, one message per event.\n\nUse
it to react to activity in an AWS account, for example to raise an alert when a
secret is read, a security group is changed or someone signs in to the console.
Set `eventSource` to read the events of one AWS service only.\n\nOutput: the
body is the CloudTrail record of the event, as JSON text. The headers give the
ID, the name (such as [...]
"group" : "AWS Cloudtrail",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -64,8 +64,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "AWS CloudWatch Metrics Sink",
- "summary" : "Sends one data point of a custom metric to Amazon CloudWatch
for each message, with the name, the value and the unit of the metric taken
from headers.",
- "description" : "Sends one data point of a custom metric to Amazon
CloudWatch for each message, with the name, the value and the unit of the
metric taken from headers.\n\nUse it to record business or technical numbers
from an integration, such as orders processed, items in stock or processing
time, and to graph them or set alarms on them in CloudWatch.\n\nInput: the body
is not used. The `metric-name` header (or `ce-metricname`) is the name of the
metric, and is required. The optiona [...]
+ "summary" : "Sends one data point of a custom metric to Amazon CloudWatch
for each message, with the name, the value and the unit of the metric taken
from headers or options.",
+ "description" : "Sends one data point of a custom metric to Amazon
CloudWatch for each message, with the name, the value and the unit of the
metric taken from headers or options.\n\nUse it to record business or technical
numbers from an integration, such as orders processed, items in stock or
processing time, and to graph them or set alarms on them in
CloudWatch.\n\nInput: the body is not used. The `metric-name` header (or
`ce-metricname`) is the name of the metric; it is required un [...]
"group" : "AWS Cloudwatch",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -95,7 +95,7 @@
"supportLevel" : "Stable",
"title" : "AWS DynamoDB Streams Source",
"summary" : "Receives the changes made to the items of an Amazon DynamoDB
table from its stream, one message per inserted, modified or removed item.",
- "description" : "Receives the changes made to the items of an Amazon
DynamoDB table from its stream, one message per inserted, modified or removed
item.\n\nUse it to react when data changes in a table, for example to copy new
orders to another system, keep a search index or a cache up to date, or record
an audit trail. To write items into a table, use `aws-ddb-sink`.\n\nOutput: the
body is a JSON object, as JSON text, that describes one change: `eventName` is
`INSERT`, `MODIFY` or `R [...]
+ "description" : "Receives the changes made to the items of an Amazon
DynamoDB table from its stream, one message per inserted, modified or removed
item.\n\nUse it to react when data changes in a table, for example to copy new
orders to another system, keep a search index or a cache up to date, or record
an audit trail. To write items into a table, use `aws-ddb-sink`.\n\nOutput: the
body is a JSON object, as JSON text, that describes one change: `eventName` is
`INSERT`, `MODIFY` or `R [...]
"group" : "AWS DynamoDB Streams",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
@@ -145,7 +145,7 @@
"supportLevel" : "Stable",
"title" : "AWS Kinesis Source",
"summary" : "Reads the records of an Amazon Kinesis data stream, one
message per record.",
- "description" : "Reads the records of an Amazon Kinesis data stream, one
message per record.\n\nUse it to process a stream of events that other
applications write to Kinesis, such as clicks, logs or readings from devices,
for example with `aws-kinesis-sink`.\n\nOutput: the body is the data of the
record, as bytes. The headers give the partition key, the sequence number, the
shard ID and the arrival time of the record.\n\nGood to know: by default it
reads the stream from its oldest re [...]
+ "description" : "Reads the records of an Amazon Kinesis data stream, one
message per record.\n\nUse it to process a stream of events that other
applications write to Kinesis, such as clicks, logs or readings from devices,
for example with `aws-kinesis-sink`.\n\nOutput: the body is the data of the
record, as bytes. The headers give the partition key, the sequence number, the
shard ID and the arrival time of the record.\n\nGood to know: by default it
reads the stream from its oldest re [...]
"group" : "AWS Kinesis",
"namespace" : "AWS",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-cloudtrail-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-cloudtrail-source.kamelet.yaml
index 6f25ad7e6..740531e97 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-cloudtrail-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-cloudtrail-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the CloudTrail record of the event, as JSON text.
The headers give the ID, the name (such as `GetSecretValue`), the source (such
as `secretsmanager.amazonaws.com`) and the user name of the event.
- Good to know: it looks up the management events that CloudTrail keeps
for the region, which go back 90 days, so when it starts it also reads events
from before. It remembers which events it read only while it runs: after a
restart it reads them again. Nothing is removed from CloudTrail. An event can
be looked up only some minutes after it happened. To reach AWS through an HTTP
proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP` for a
proxy that takes plain HTT [...]
+ Good to know: it looks up the management events that CloudTrail keeps
for the region, and reads the events that happen after it starts. It remembers
where it is only while it runs: the events that happen while it is stopped are
not read. Nothing is removed from CloudTrail. An event can be looked up only
some minutes after it happened, and sometimes after a newer event: every
`lookBackInterval` it looks up the last `lookBackWindow` again, and reads the
events it missed. To reach AWS [...]
required:
- region
type: object
@@ -110,6 +110,16 @@ spec:
description: Read only the events of this AWS service, given as its
address, such as `s3.amazonaws.com` or `secretsmanager.amazonaws.com`. When it
is not set, the events of all services are read.
type: string
example: secretsmanager.amazonaws.com
+ lookBackWindow:
+ title: Look Back Window
+ description: How far back it looks up the events again, in
milliseconds, to read the events that CloudTrail made visible after a newer
one. When it is not set, it is 600000, ten minutes. 0 turns the second lookup
off, and then such a late event is not read. Needs Camel 4.24 or later.
+ type: integer
+ example: 600000
+ lookBackInterval:
+ title: Look Back Interval
+ description: How often it looks up the `lookBackWindow` again, in
milliseconds. A late event is read at most this long after it became visible.
When it is not set, it is 60000, one minute. It must be above 0 unless
`lookBackWindow` is 0, or the integration does not start. Needs Camel 4.24 or
later.
+ type: integer
+ example: 60000
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -200,6 +210,8 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ lookBackWindow: "{{?lookBackWindow}}"
+ lookBackInterval: "{{?lookBackInterval}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-cloudwatch-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-cloudwatch-sink.kamelet.yaml
index 905463ee8..b6da2a46a 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-cloudwatch-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-cloudwatch-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "AWS CloudWatch Metrics Sink"
description: |-
- Sends one data point of a custom metric to Amazon CloudWatch for each
message, with the name, the value and the unit of the metric taken from headers.
+ Sends one data point of a custom metric to Amazon CloudWatch for each
message, with the name, the value and the unit of the metric taken from headers
or options.
Use it to record business or technical numbers from an integration, such
as orders processed, items in stock or processing time, and to graph them or
set alarms on them in CloudWatch.
- Input: the body is not used. The `metric-name` header (or
`ce-metricname`) is the name of the metric, and is required. The optional
`metric-value` header (or `ce-metricvalue`) is the value, a number, 1 when it
is not set. The optional `metric-unit` header (or `ce-metricunit`) is the unit,
such as `Count`, `Seconds`, `Bytes` or `Percent`, `Count` when it is not set.
The optional `metric-dimension-name` and `metric-dimension-value` headers (or
`ce-metricdimensionname` and `ce-metricd [...]
+ Input: the body is not used. The `metric-name` header (or
`ce-metricname`) is the name of the metric; it is required unless the
`metricName` option is set. The optional `metric-value` header (or
`ce-metricvalue`) is the value, a number; when it is not set, the value is
`metricValue`, or 1. The optional `metric-unit` header (or `ce-metricunit`) is
the unit, such as `Count`, `Seconds`, `Bytes` or `Percent`; when it is not set,
the unit is `metricUnit`, or `Count`. A header wins over [...]
Output: the message is unchanged.
- Good to know: a message without a metric name, or with a value that is
not a number, fails. The time in `metric-timestamp` must be a Java `Instant` or
`Date`: text is ignored, and then the data point gets the time it is sent. To
reach AWS through an HTTP proxy, set `proxyHost` and `proxyPort`, and set
`proxyProtocol` to `HTTP` for a proxy that takes plain HTTP, such as Squid.
+ Good to know: a message without a metric name, when `metricName` is not
set either, fails, and so does a value that is not a number or a unit that
CloudWatch does not know. Set the options to send the same metric for every
message, for example `metricName` `OrdersProcessed` to count the messages. The
time in `metric-timestamp` must be a Java `Instant` or `Date`: text is ignored,
and then the data point gets the time it is sent. To reach AWS through an HTTP
proxy, set `proxyHost` an [...]
required:
- cwNamespace
- region
@@ -51,6 +51,23 @@ spec:
description: The namespace that groups your metrics in CloudWatch,
such as `MyApp/Orders`. It must not start with `AWS/`, which is kept for the
metrics of AWS services.
example: MyApp/Orders
type: string
+ metricName:
+ title: Metric Name
+ description: The name of the metric, such as `OrdersProcessed`, for
the messages without a `metric-name` header. Up to 255 characters, which can be
letters, digits, spaces and punctuation. When neither is set, the message fails.
+ type: string
+ pattern: "^[ -~]{1,255}$"
+ example: OrdersProcessed
+ metricValue:
+ title: Metric Value
+ description: The value of the data point, a number, for the messages
without a `metric-value` header. When neither is set, the value is 1.
+ type: number
+ example: 1
+ metricUnit:
+ title: Metric Unit
+ description: The unit of the value, for the messages without a
`metric-unit` header. When neither is set, the unit is `Count`.
+ type: string
+ enum: ["Seconds", "Microseconds", "Milliseconds", "Bytes",
"Kilobytes", "Megabytes", "Gigabytes", "Terabytes", "Bits", "Kilobits",
"Megabits", "Gigabits", "Terabits", "Percent", "Count", "Bytes/Second",
"Kilobytes/Second", "Megabytes/Second", "Gigabytes/Second", "Terabytes/Second",
"Bits/Second", "Kilobits/Second", "Megabits/Second", "Gigabits/Second",
"Terabits/Second", "Count/Second", "None"]
+ example: Milliseconds
accessKey:
title: Access Key
description: The access key ID of an IAM user, which you create in the
AWS console under IAM, Users, Security credentials. Use it together with
`secretKey`. Leave both out when `useDefaultCredentialsProvider` or
`useProfileCredentialsProvider` is true.
@@ -130,15 +147,15 @@ spec:
headers:
metric-name:
title: Metric Name
- description: The name of the metric, such as `OrdersProcessed`.
`ce-metricname` works too. Required.
+ description: The name of the metric, such as `OrdersProcessed`.
`ce-metricname` works too. Required unless the `metricName` option is set.
type: string
metric-value:
title: Metric Value
- description: The value of the data point, a number. `ce-metricvalue`
works too. 1 when it is not set.
+ description: The value of the data point, a number. `ce-metricvalue`
works too. When it is not set, the value is the `metricValue` option, or 1.
type: number
metric-unit:
title: Metric Unit
- description: The unit of the value, such as `Count`, `Seconds`,
`Bytes` or `Percent`. `ce-metricunit` works too. `Count` when it is not set.
+ description: The unit of the value, such as `Count`, `Seconds`,
`Bytes` or `Percent`. `ce-metricunit` works too. When it is not set, the unit
is the `metricUnit` option, or `Count`.
type: string
metric-dimension-name:
title: Dimension Name
@@ -296,3 +313,6 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ name: "RAW({{?metricName}})"
+ value: "{{?metricValue}}"
+ unit: "{{?metricUnit}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-ddb-streams-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-ddb-streams-source.kamelet.yaml
index 6002fc0d5..56c4190df 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-ddb-streams-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-ddb-streams-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is a JSON object, as JSON text, that describes one
change: `eventName` is `INSERT`, `MODIFY` or `REMOVE`, and `dynamodb` holds the
key of the item (`keys`) and, depending on the view type of the stream, the
item before (`oldImage`) and after (`newImage`) the change, with the type of
each attribute. The headers give the ID and the name of the event.
- Good to know: DynamoDB Streams must be enabled on the table, with the
view type that holds the images you need, such as new and old images. By
default it reads only the changes made after it starts. To reach AWS through an
HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP`
for a proxy that takes plain HTTP, such as Squid.
+ Good to know: DynamoDB Streams must be enabled on the table, with the
view type that holds the images you need, such as new and old images. By
default it reads only the changes made after it starts, up to 100 in one poll;
set `maxResultsPerRequest` to read fewer at a time. To reach AWS through an
HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol` to `HTTP`
for a proxy that takes plain HTTP, such as Squid.
required:
- table
- region
@@ -116,6 +116,11 @@ spec:
description: How long to wait between two reads of the stream, in
milliseconds.
type: integer
default: 500
+ maxResultsPerRequest:
+ title: Max Changes Per Poll
+ description: The most changes it reads in one poll, from 1 to 1000.
The number is shared between the shards of the stream that it reads, at least 1
each. When it is not set, it is 100. With a lower number, a burst of changes
takes more polls, `delay` apart, to read.
+ type: integer
+ example: 50
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -203,6 +208,7 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxResultsPerRequest: "{{?maxResultsPerRequest}}"
steps:
- transformDataType:
toType: aws2-ddb:application-x-struct
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
index 775e05694..e48136fca 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is the data of the record, as bytes. The headers give
the partition key, the sequence number, the shard ID and the arrival time of
the record.
- Good to know: by default it reads the stream from its oldest record, and
again from the oldest record each time the integration starts: it does not
remember where it stopped, so records are read again after a restart. Set
`iteratorType` to `LATEST` to read only the records written after it started;
the records written while it was stopped are then not read. `iteratorType` can
also start at a sequence number or at a time, again on each start. Records are
not removed: Kinesis keeps t [...]
+ Good to know: by default it reads the stream from its oldest record, and
again from the oldest record each time the integration starts: it does not
remember where it stopped, so records are read again after a restart. Set
`iteratorType` to `LATEST` to read only the records written after it started;
the records written while it was stopped are then not read. `iteratorType` can
also start at a sequence number or at a time, again on each start. Records are
not removed: Kinesis keeps t [...]
required:
- stream
- region
@@ -128,6 +128,23 @@ spec:
description: How long to wait between two polls of the stream, in
milliseconds.
type: integer
default: 500
+ maxResultsPerRequest:
+ title: Max Records Per Poll
+ description: The most records it reads from each shard in one poll,
from 1 to 10000. When it is not set, it is 1000. Not used when
`useKclConsumers` is true.
+ type: integer
+ example: 100
+ shardId:
+ title: Shard ID
+ description: The ID of the one shard to read, such as
`shardId-000000000000`, as the AWS console or `aws kinesis list-shards` shows
it. When it is not set, it reads all the shards of the stream. Not used when
`useKclConsumers` is true.
+ type: string
+ pattern: "^shardId-[0-9]+$"
+ example: shardId-000000000000
+ shardClosed:
+ title: Closed Shard
+ description: "What to do when it has read the last record of a closed
shard. A shard is closed when the stream is resharded, for example when the
number of shards is changed, and new shards take its place. `ignore` logs a
warning once and goes on with the other shards, and so does `silent`, without
the warning. With `fail`, each later poll fails with an error and a stack trace
in the log, for as long as the closed shard is listed, which is 24 hours by
default; the records of the [...]
+ type: string
+ enum: ["ignore", "silent", "fail"]
+ example: ignore
asyncClient:
title: Async Client
description: If true, it uses the asynchronous Kinesis client of the
AWS SDK. `useKclConsumers` needs it.
@@ -143,6 +160,12 @@ spec:
description: If true, the Kinesis Client Library does not send its
metrics to CloudWatch. Only used when `useKclConsumers` is true.
type: boolean
default: false
+ applicationName:
+ title: KCL Application Name
+ description: "The name of the application for the Kinesis Client
Library. The KCL keeps where it stopped, and which copy of the integration
reads which shard, in a DynamoDB table of this name, which it creates. Copies
of the integration with the same name share the shards of the stream; give
another integration that reads the same stream its own name, so that it reads
all the records too. When it is not set, it is the name of the stream. Only
used when `useKclConsumers` is true."
+ type: string
+ pattern: "^[A-Za-z0-9_.-]{3,255}$"
+ example: orders-processor
proxyHost:
title: Proxy Host
description: The host name or IP address of an HTTP proxy to send the
calls to AWS through, such as the proxy of a company network. Used only
together with `proxyPort`. When it is not set, the calls go straight to AWS.
@@ -210,6 +233,10 @@ spec:
proxyHost: "{{?proxyHost}}"
proxyPort: "{{?proxyPort}}"
proxyProtocol: "{{?proxyProtocol}}"
+ maxResultsPerRequest: "{{?maxResultsPerRequest}}"
+ shardId: "{{?shardId}}"
+ shardClosed: "{{?shardClosed}}"
+ applicationName: "{{?applicationName}}"
steps:
- to:
uri: kamelet:sink