shashank created CAMEL-26968:
--------------------------------

             Summary: camel-aws2-kinesis - iteratorType LATEST: when the 
iterator of a shard expires before a record of it was routed, a new LATEST 
iterator skips the records stored in between
                 Key: CAMEL-26968
                 URL: https://issues.apache.org/jira/browse/CAMEL-26968
             Project: Camel
          Issue Type: Bug
          Components: camel-aws2-kinesis
            Reporter: shashank
            Assignee: shashank


Found by the Kinesis DST world of CAMEL-26948 (PR #29993, branch 
{{camel-aws2-kinesis-dst-world-v2}}): its default set accepts this case, 
{{-Ddst.kinesis.strictLatest=true}} counts the records as missing and fails 44 
of 2000 seeds. Reproduced outside the DST harness with a plain unit test 
(below). It is the case CAMEL-25569 left open.

h3. What happens

Since CAMEL-25569, when a cached shard iterator has expired, 
{{Kinesis2Consumer.getShardIterator}} asks for {{AFTER_SEQUENCE_NUMBER}} with 
the last record it routed from that shard. A shard from which no record was 
routed yet has no such sequence number, and the consumer asks for the 
configured {{iteratorType}} again. With {{LATEST}} that is a new start point: 
the records stored in the shard between the previous iterator and the new one 
are skipped, while the code comment next to it says this is what the 
AFTER_SEQUENCE_NUMBER branch avoids ("skip the new ones (LATEST)").

An iterator expires when the consumer does not use it for 5 minutes: the route 
was suspended for longer, {{delay}} or the back-off is longer, or the route 
took that long for the records of another shard in the same poll. A shard that 
gets few records, or none for a while, is the typical case: its iterators keep 
returning no record, so the consumer has no sequence number for it.

h3. Expected

GetShardIterator: "LATEST - Start reading just after the most recent record in 
the shard"; "AT_TIMESTAMP - Start reading from the position denoted by a 
specific time stamp"; "A shard iterator expires 5 minutes after it is returned 
to the requester". {{iteratorType=LATEST}} is where the consumer starts a 
shard. Once started, an expired iterator should not move that start: the 
records stored after the first LATEST iterator are new for the consumer and 
should be routed, as they are when a record was routed before the expiry.

h3. Reproduction

New test in {{KinesisConsumerPositionTest}} (the in-memory stream of that test, 
now with arrival timestamps and AT_TIMESTAMP): LATEST, the first poll gets no 
record, a is stored, the iterators expire, b is stored. On main the next poll 
routes nothing (a and b are skipped); expected a and b.

h3. Fix

The consumer remembers, per shard, the time just before it asked for its first 
LATEST iterator. When the iterator of such a shard expires before a record of 
it was routed, it asks for {{AT_TIMESTAMP}} with that time (the resume 
strategy, if any, can still change the request as before). No record routed yet 
means no duplicate; the records stored since the start are routed. The time 
comes from the clock of the consumer, so a clock ahead of AWS by s seconds can 
still miss the records of those s seconds, and one behind it can route a few 
records stored just before the start. Branch 
{{camel-aws2-kinesis-latest-iterator-expiry}}.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to