shashank created CAMEL-26968:
--------------------------------
Summary: camel-aws2-kinesis - iteratorType LATEST: when the
iterator of a shard expires before a record of it was routed, a new LATEST
iterator skips the records stored in between
Key: CAMEL-26968
URL: https://issues.apache.org/jira/browse/CAMEL-26968
Project: Camel
Issue Type: Bug
Components: camel-aws2-kinesis
Reporter: shashank
Assignee: shashank
Found by the Kinesis DST world of CAMEL-26948 (PR #29993, branch
{{camel-aws2-kinesis-dst-world-v2}}): its default set accepts this case,
{{-Ddst.kinesis.strictLatest=true}} counts the records as missing and fails 44
of 2000 seeds. Reproduced outside the DST harness with a plain unit test
(below). It is the case CAMEL-25569 left open.
h3. What happens
Since CAMEL-25569, when a cached shard iterator has expired,
{{Kinesis2Consumer.getShardIterator}} asks for {{AFTER_SEQUENCE_NUMBER}} with
the last record it routed from that shard. A shard from which no record was
routed yet has no such sequence number, and the consumer asks for the
configured {{iteratorType}} again. With {{LATEST}} that is a new start point:
the records stored in the shard between the previous iterator and the new one
are skipped, while the code comment next to it says this is what the
AFTER_SEQUENCE_NUMBER branch avoids ("skip the new ones (LATEST)").
An iterator expires when the consumer does not use it for 5 minutes: the route
was suspended for longer, {{delay}} or the back-off is longer, or the route
took that long for the records of another shard in the same poll. A shard that
gets few records, or none for a while, is the typical case: its iterators keep
returning no record, so the consumer has no sequence number for it.
h3. Expected
GetShardIterator: "LATEST - Start reading just after the most recent record in
the shard"; "AT_TIMESTAMP - Start reading from the position denoted by a
specific time stamp"; "A shard iterator expires 5 minutes after it is returned
to the requester". {{iteratorType=LATEST}} is where the consumer starts a
shard. Once started, an expired iterator should not move that start: the
records stored after the first LATEST iterator are new for the consumer and
should be routed, as they are when a record was routed before the expiry.
h3. Reproduction
New test in {{KinesisConsumerPositionTest}} (the in-memory stream of that test,
now with arrival timestamps and AT_TIMESTAMP): LATEST, the first poll gets no
record, a is stored, the iterators expire, b is stored. On main the next poll
routes nothing (a and b are skipped); expected a and b.
h3. Fix
The consumer remembers, per shard, the time just before it asked for its first
LATEST iterator. When the iterator of such a shard expires before a record of
it was routed, it asks for {{AT_TIMESTAMP}} with that time (the resume
strategy, if any, can still change the request as before). No record routed yet
means no duplicate; the records stored since the start are routed. The time
comes from the clock of the consumer, so a clock ahead of AWS by s seconds can
still miss the records of those s seconds, and one behind it can route a few
records stored just before the start. Branch
{{camel-aws2-kinesis-latest-iterator-expiry}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)