[ 
https://issues.apache.org/jira/browse/CAMEL-26968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-26968 started by shashank.
----------------------------------------
> camel-aws2-kinesis - iteratorType LATEST: when the iterator of a shard 
> expires before a record of it was routed, a new LATEST iterator skips the 
> records stored in between
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-26968
>                 URL: https://issues.apache.org/jira/browse/CAMEL-26968
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-aws2-kinesis
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> Found by the Kinesis DST world of CAMEL-26948 (PR #29993, branch 
> {{camel-aws2-kinesis-dst-world-v2}}): its default set accepts this case, 
> {{-Ddst.kinesis.strictLatest=true}} counts the records as missing and fails 
> 44 of 2000 seeds. Reproduced outside the DST harness with a plain unit test 
> (below). It is the case CAMEL-25569 left open.
> h3. What happens
> Since CAMEL-25569, when a cached shard iterator has expired, 
> {{Kinesis2Consumer.getShardIterator}} asks for {{AFTER_SEQUENCE_NUMBER}} with 
> the last record it routed from that shard. A shard from which no record was 
> routed yet has no such sequence number, and the consumer asks for the 
> configured {{iteratorType}} again. With {{LATEST}} that is a new start point: 
> the records stored in the shard between the previous iterator and the new one 
> are skipped, while the code comment next to it says this is what the 
> AFTER_SEQUENCE_NUMBER branch avoids ("skip the new ones (LATEST)").
> An iterator expires when the consumer does not use it for 5 minutes: the 
> route was suspended for longer, {{delay}} or the back-off is longer, or the 
> route took that long for the records of another shard in the same poll. A 
> shard that gets few records, or none for a while, is the typical case: its 
> iterators keep returning no record, so the consumer has no sequence number 
> for it.
> h3. Expected
> GetShardIterator: "LATEST - Start reading just after the most recent record 
> in the shard"; "AT_TIMESTAMP - Start reading from the position denoted by a 
> specific time stamp"; "A shard iterator expires 5 minutes after it is 
> returned to the requester". {{iteratorType=LATEST}} is where the consumer 
> starts a shard. Once started, an expired iterator should not move that start: 
> the records stored after the first LATEST iterator are new for the consumer 
> and should be routed, as they are when a record was routed before the expiry.
> h3. Reproduction
> New test in {{KinesisConsumerPositionTest}} (the in-memory stream of that 
> test, now with arrival timestamps and AT_TIMESTAMP): LATEST, the first poll 
> gets no record, a is stored, the iterators expire, b is stored. On main the 
> next poll routes nothing (a and b are skipped); expected a and b.
> h3. Fix
> The consumer remembers, per shard, the time just before it asked for its 
> first LATEST iterator. When the iterator of such a shard expires before a 
> record of it was routed, it asks for {{AT_TIMESTAMP}} with that time (the 
> resume strategy, if any, can still change the request as before). No record 
> routed yet means no duplicate; the records stored since the start are routed. 
> The time comes from the clock of the consumer, so a clock ahead of AWS by s 
> seconds can still miss the records of those s seconds, and one behind it can 
> route a few records stored just before the start. Branch 
> {{camel-aws2-kinesis-latest-iterator-expiry}}.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to