[
https://issues.apache.org/jira/browse/CAMEL-26968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26968 started by shashank.
----------------------------------------
> camel-aws2-kinesis - iteratorType LATEST: when the iterator of a shard
> expires before a record of it was routed, a new LATEST iterator skips the
> records stored in between
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26968
> URL: https://issues.apache.org/jira/browse/CAMEL-26968
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2-kinesis
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found by the Kinesis DST world of CAMEL-26948 (PR #29993, branch
> {{camel-aws2-kinesis-dst-world-v2}}): its default set accepts this case,
> {{-Ddst.kinesis.strictLatest=true}} counts the records as missing and fails
> 44 of 2000 seeds. Reproduced outside the DST harness with a plain unit test
> (below). It is the case CAMEL-25569 left open.
> h3. What happens
> Since CAMEL-25569, when a cached shard iterator has expired,
> {{Kinesis2Consumer.getShardIterator}} asks for {{AFTER_SEQUENCE_NUMBER}} with
> the last record it routed from that shard. A shard from which no record was
> routed yet has no such sequence number, and the consumer asks for the
> configured {{iteratorType}} again. With {{LATEST}} that is a new start point:
> the records stored in the shard between the previous iterator and the new one
> are skipped, while the code comment next to it says this is what the
> AFTER_SEQUENCE_NUMBER branch avoids ("skip the new ones (LATEST)").
> An iterator expires when the consumer does not use it for 5 minutes: the
> route was suspended for longer, {{delay}} or the back-off is longer, or the
> route took that long for the records of another shard in the same poll. A
> shard that gets few records, or none for a while, is the typical case: its
> iterators keep returning no record, so the consumer has no sequence number
> for it.
> h3. Expected
> GetShardIterator: "LATEST - Start reading just after the most recent record
> in the shard"; "AT_TIMESTAMP - Start reading from the position denoted by a
> specific time stamp"; "A shard iterator expires 5 minutes after it is
> returned to the requester". {{iteratorType=LATEST}} is where the consumer
> starts a shard. Once started, an expired iterator should not move that start:
> the records stored after the first LATEST iterator are new for the consumer
> and should be routed, as they are when a record was routed before the expiry.
> h3. Reproduction
> New test in {{KinesisConsumerPositionTest}} (the in-memory stream of that
> test, now with arrival timestamps and AT_TIMESTAMP): LATEST, the first poll
> gets no record, a is stored, the iterators expire, b is stored. On main the
> next poll routes nothing (a and b are skipped); expected a and b.
> h3. Fix
> The consumer remembers, per shard, the time just before it asked for its
> first LATEST iterator. When the iterator of such a shard expires before a
> record of it was routed, it asks for {{AT_TIMESTAMP}} with that time (the
> resume strategy, if any, can still change the request as before). No record
> routed yet means no duplicate; the records stored since the start are routed.
> The time comes from the clock of the consumer, so a clock ahead of AWS by s
> seconds can still miss the records of those s seconds, and one behind it can
> route a few records stored just before the start. Branch
> {{camel-aws2-kinesis-latest-iterator-expiry}}.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)