This is an automated email from the ASF dual-hosted git repository.

oscerd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git


The following commit(s) were added to refs/heads/main by this push:
     new ac6606484 chore: aws-kinesis-source can choose where it starts reading 
(#3170) (#3172)
ac6606484 is described below

commit ac660648450ca927f84324c9c6a0da8bdbb6fd3c
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 09:53:40 2026 +0100

    chore: aws-kinesis-source can choose where it starts reading (#3170) (#3172)
    
    * chore: aws-kinesis-source can start at the latest record (#3170)
    
    Add the iteratorType option (TRIM_HORIZON, LATEST, AT_SEQUENCE_NUMBER,
    AFTER_SEQUENCE_NUMBER, AT_TIMESTAMP) and its companions sequenceNumber
    and messageTimestamp. The default, TRIM_HORIZON, is the default of the
    aws2-kinesis component, so the Kamelet reads as before when they are
    not set. The description says the start position is not remembered
    across restarts, and that the KCL consumer does not use iteratorType.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * chore: aws-kinesis-source: keep only checked facts in the sequenceNumber 
description (#3170)
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    ---------
    
    Co-authored-by: Claude Opus 5.5 <[email protected]>
---
 .../partials/aws-kinesis-source-description.adoc    | 21 +++++++++++++++++++++
 kamelets/aws-kinesis-source.kamelet.yaml            | 21 ++++++++++++++++++++-
 .../src/main/resources/META-INF/kamelets-index.json |  2 +-
 .../kamelets/aws-kinesis-source.kamelet.yaml        | 21 ++++++++++++++++++++-
 4 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc 
b/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
index 6c3340a59..13f2ce353 100644
--- a/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
+++ b/docs/modules/ROOT/partials/aws-kinesis-source-description.adoc
@@ -37,6 +37,27 @@ You could consume the stream content directly
             uri: "kamelet:log-sink"
 ----
 
+=== Where it starts reading
+
+By default (`iteratorType` is `TRIM_HORIZON`) the Kamelet reads the stream 
from its oldest record each time the integration starts, so the records are 
read again after a restart. To read only the records written after the 
integration started, set `iteratorType` to `LATEST`:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+    from:
+      uri: "kamelet:aws-kinesis-source"
+      parameters:
+        useDefaultCredentialsProvider: true
+        region: "eu-west-1"
+        stream: "kamelets"
+        iteratorType: "LATEST"
+      steps:
+        - to:
+            uri: "kamelet:log-sink"
+----
+
+The records written while the integration was stopped are then not read. To 
start at a given record, set `iteratorType` to `AT_SEQUENCE_NUMBER` or 
`AFTER_SEQUENCE_NUMBER` and `sequenceNumber` to its sequence number; to start 
at a given time, set `iteratorType` to `AT_TIMESTAMP` and `messageTimestamp` to 
the time, such as `2026-11-17T08:00:00Z`. The Kamelet does not remember where 
it stopped: it starts at the same place on each start. The KCL consumer below 
does not use `iteratorType`.
+
 === Usage example with KCL Consumer
 
 You could consume the stream content with the KCL support
diff --git a/kamelets/aws-kinesis-source.kamelet.yaml 
b/kamelets/aws-kinesis-source.kamelet.yaml
index 2a943163b..be73ea5b0 100644
--- a/kamelets/aws-kinesis-source.kamelet.yaml
+++ b/kamelets/aws-kinesis-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
 
       Output: the body is the data of the record, as bytes. The headers give 
the partition key, the sequence number, the shard ID and the arrival time of 
the record.
 
-      Good to know: it reads the stream from its oldest record, and again from 
the oldest record each time the integration starts: it does not remember where 
it stopped, so records are read again after a restart. Records are not removed: 
Kinesis keeps them for the retention period of the stream. With 
`useKclConsumers` it reads through the Kinesis Client Library instead, and the 
body is a `ByteBuffer` rather than bytes.
+      Good to know: by default it reads the stream from its oldest record, and 
again from the oldest record each time the integration starts: it does not 
remember where it stopped, so records are read again after a restart. Set 
`iteratorType` to `LATEST` to read only the records written after it started; 
the records written while it was stopped are then not read. `iteratorType` can 
also start at a sequence number or at a time, again on each start. Records are 
not removed: Kinesis keeps t [...]
     required:
       - stream
       - region
@@ -107,6 +107,22 @@ spec:
         description: If true, it connects to the address in 
`uriEndpointOverride` instead of to AWS. Use it for a local stand-in of AWS 
such as LocalStack.
         type: boolean
         default: false
+      iteratorType:
+        title: Start Position
+        description: "Where in the stream it starts reading each time the 
integration starts. `TRIM_HORIZON` starts at the oldest record the stream still 
keeps, so the records are read again after each restart. `LATEST` reads only 
the records written after it started. `AT_SEQUENCE_NUMBER` and 
`AFTER_SEQUENCE_NUMBER` start at, or just after, the record whose sequence 
number is in `sequenceNumber`. `AT_TIMESTAMP` starts at the first record that 
arrived at or after the time in `messageTimes [...]
+        type: string
+        enum: ["TRIM_HORIZON", "LATEST", "AT_SEQUENCE_NUMBER", 
"AFTER_SEQUENCE_NUMBER", "AT_TIMESTAMP"]
+        default: TRIM_HORIZON
+      sequenceNumber:
+        title: Sequence Number
+        description: The sequence number of the record to start at, as Kinesis 
gives it, such as the value of the `CamelAwsKinesisSequenceNumber` header of a 
record read earlier. Required, and only used, when `iteratorType` is 
`AT_SEQUENCE_NUMBER` or `AFTER_SEQUENCE_NUMBER`.
+        example: "49655467183424123548936502291338416434578112145614569474"
+        type: string
+      messageTimestamp:
+        title: Start Time
+        description: The time to start reading at, either as an ISO 8601 date 
and time in UTC, such as `2026-11-17T08:00:00Z`, or as milliseconds since 
1970-01-01 UTC, such as `1794902400000`. It starts at the first record that 
arrived in the stream at or after this time. Required, and only used, when 
`iteratorType` is `AT_TIMESTAMP`.
+        example: "2026-11-17T08:00:00Z"
+        type: string
       delay:
         title: Delay
         description: How long to wait between two polls of the stream, in 
milliseconds.
@@ -160,6 +176,9 @@ spec:
         accessKey: "RAW({{?accessKey}})"
         asyncClient: "{{asyncClient}}"
         delay: "{{delay}}"
+        iteratorType: "{{iteratorType}}"
+        sequenceNumber: "{{?sequenceNumber}}"
+        messageTimestamp: "{{?messageTimestamp}}"
         kclDisableCloudwatchMetricsExport: 
"{{kclDisableCloudwatchMetricsExport}}"
         overrideEndpoint: "{{overrideEndpoint}}"
         profileCredentialsName: "{{?profileCredentialsName}}"
diff --git 
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json 
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 20b3569fc..0a1992a28 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -145,7 +145,7 @@
     "supportLevel" : "Stable",
     "title" : "AWS Kinesis Source",
     "summary" : "Reads the records of an Amazon Kinesis data stream, one 
message per record.",
-    "description" : "Reads the records of an Amazon Kinesis data stream, one 
message per record.\n\nUse it to process a stream of events that other 
applications write to Kinesis, such as clicks, logs or readings from devices, 
for example with `aws-kinesis-sink`.\n\nOutput: the body is the data of the 
record, as bytes. The headers give the partition key, the sequence number, the 
shard ID and the arrival time of the record.\n\nGood to know: it reads the 
stream from its oldest record, and a [...]
+    "description" : "Reads the records of an Amazon Kinesis data stream, one 
message per record.\n\nUse it to process a stream of events that other 
applications write to Kinesis, such as clicks, logs or readings from devices, 
for example with `aws-kinesis-sink`.\n\nOutput: the body is the data of the 
record, as bytes. The headers give the partition key, the sequence number, the 
shard ID and the arrival time of the record.\n\nGood to know: by default it 
reads the stream from its oldest re [...]
     "group" : "AWS Kinesis",
     "namespace" : "AWS",
     "provider" : "Apache Software Foundation"
diff --git 
a/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
 
b/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
index 2a943163b..be73ea5b0 100644
--- 
a/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
+++ 
b/library/camel-kamelets/src/main/resources/kamelets/aws-kinesis-source.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
 
       Output: the body is the data of the record, as bytes. The headers give 
the partition key, the sequence number, the shard ID and the arrival time of 
the record.
 
-      Good to know: it reads the stream from its oldest record, and again from 
the oldest record each time the integration starts: it does not remember where 
it stopped, so records are read again after a restart. Records are not removed: 
Kinesis keeps them for the retention period of the stream. With 
`useKclConsumers` it reads through the Kinesis Client Library instead, and the 
body is a `ByteBuffer` rather than bytes.
+      Good to know: by default it reads the stream from its oldest record, and 
again from the oldest record each time the integration starts: it does not 
remember where it stopped, so records are read again after a restart. Set 
`iteratorType` to `LATEST` to read only the records written after it started; 
the records written while it was stopped are then not read. `iteratorType` can 
also start at a sequence number or at a time, again on each start. Records are 
not removed: Kinesis keeps t [...]
     required:
       - stream
       - region
@@ -107,6 +107,22 @@ spec:
         description: If true, it connects to the address in 
`uriEndpointOverride` instead of to AWS. Use it for a local stand-in of AWS 
such as LocalStack.
         type: boolean
         default: false
+      iteratorType:
+        title: Start Position
+        description: "Where in the stream it starts reading each time the 
integration starts. `TRIM_HORIZON` starts at the oldest record the stream still 
keeps, so the records are read again after each restart. `LATEST` reads only 
the records written after it started. `AT_SEQUENCE_NUMBER` and 
`AFTER_SEQUENCE_NUMBER` start at, or just after, the record whose sequence 
number is in `sequenceNumber`. `AT_TIMESTAMP` starts at the first record that 
arrived at or after the time in `messageTimes [...]
+        type: string
+        enum: ["TRIM_HORIZON", "LATEST", "AT_SEQUENCE_NUMBER", 
"AFTER_SEQUENCE_NUMBER", "AT_TIMESTAMP"]
+        default: TRIM_HORIZON
+      sequenceNumber:
+        title: Sequence Number
+        description: The sequence number of the record to start at, as Kinesis 
gives it, such as the value of the `CamelAwsKinesisSequenceNumber` header of a 
record read earlier. Required, and only used, when `iteratorType` is 
`AT_SEQUENCE_NUMBER` or `AFTER_SEQUENCE_NUMBER`.
+        example: "49655467183424123548936502291338416434578112145614569474"
+        type: string
+      messageTimestamp:
+        title: Start Time
+        description: The time to start reading at, either as an ISO 8601 date 
and time in UTC, such as `2026-11-17T08:00:00Z`, or as milliseconds since 
1970-01-01 UTC, such as `1794902400000`. It starts at the first record that 
arrived in the stream at or after this time. Required, and only used, when 
`iteratorType` is `AT_TIMESTAMP`.
+        example: "2026-11-17T08:00:00Z"
+        type: string
       delay:
         title: Delay
         description: How long to wait between two polls of the stream, in 
milliseconds.
@@ -160,6 +176,9 @@ spec:
         accessKey: "RAW({{?accessKey}})"
         asyncClient: "{{asyncClient}}"
         delay: "{{delay}}"
+        iteratorType: "{{iteratorType}}"
+        sequenceNumber: "{{?sequenceNumber}}"
+        messageTimestamp: "{{?messageTimestamp}}"
         kclDisableCloudwatchMetricsExport: 
"{{kclDisableCloudwatchMetricsExport}}"
         overrideEndpoint: "{{overrideEndpoint}}"
         profileCredentialsName: "{{?profileCredentialsName}}"

Reply via email to