allthingssecurity opened a new pull request, #30133:
URL: https://github.com/apache/camel/pull/30133

   # Description
   
   [CAMEL-26968](https://issues.apache.org/jira/browse/CAMEL-26968)
   
   Follow-up of #27753 (CAMEL-25569), for the case it left open. Found by the 
Kinesis DST world of #29993 (CAMEL-26948): its default set accepts this case, 
and `-Ddst.kinesis.strictLatest=true` fails 44 of 2000 seeds. Reproduced here 
with a plain unit test, outside the DST harness.
   
   Since CAMEL-25569, when a cached shard iterator has expired, 
`Kinesis2Consumer.getShardIterator` asks for `AFTER_SEQUENCE_NUMBER` with the 
last record routed from the shard. A shard from which no record was routed yet 
has no such sequence number, so the consumer asked for the configured 
`iteratorType` again. With `LATEST` that is a new start point, and the records 
stored between the first iterator and the new one were skipped. An iterator 
expires when it is not used for 5 minutes (a route suspended for longer, a long 
`delay` or back-off, or a poll that waits for a slow route on another shard), 
and a shard that gets few records is the typical case, as its iterators keep 
returning nothing.
   
   `iteratorType=LATEST` is where the consumer starts a shard ("Start reading 
just after the most recent record in the shard"); once started, an expired 
iterator should not move that start, as it does not once a record was routed.
   
   Change, in `Kinesis2Consumer.getShardIterator`:
   - when a request for a new iterator ends up as `LATEST` (after the resume 
strategy had its say), the consumer remembers the time just before the request, 
once per shard;
   - when the iterator of such a shard expires before a record of it was 
routed, the consumer asks for `AT_TIMESTAMP` with that time ("Start reading 
from the position denoted by a specific time stamp"). Nothing was routed from 
the shard, so nothing is routed twice; the records stored since the start are 
routed. A resume strategy can still change the request, as before;
   - the other cases are unchanged: after a routed record 
`AFTER_SEQUENCE_NUMBER`, a child shard of a resharding `TRIM_HORIZON`, the 
other iterator types the configured one (asking for it again does not move 
their start).
   
   The time comes from the clock of the consumer and Kinesis compares it with 
the arrival time it gave each record: with a clock ahead of AWS by s seconds 
the records of those s seconds can still be missed, and with one behind it a 
few records stored just before the start can be routed. Both are much smaller 
than the gap a new LATEST iterator skipped.
   
   Test, new 
`expiredIteratorOfAShardWithoutRoutedRecordsDoesNotSkipTheRecordsOfTheGapWithLatest`
 in `KinesisConsumerPositionTest` (its in-memory stream now gives each record 
an arrival timestamp and answers `AT_TIMESTAMP`): LATEST, the first poll gets 
nothing, a is stored, the iterators expire, b is stored; the next poll must 
route a and b, with an `AT_TIMESTAMP` request; after a routed record the next 
expiry uses `AFTER_SEQUENCE_NUMBER` again.
   
   Without the change the new test fails (the poll routes nothing); with it the 
camel-aws2-kinesis unit tests pass (64 run, 0 failures). With the change merged 
locally into the world of #29993, `-Ddst.kinesis.strictLatest=true` passes 2000 
of 2000 seeds (44 fail without it), and the default set passes 2000 of 2000. 
The ITs need LocalStack and were not run locally (CI runs them). The local 
offline build used the cached AWS SDK 2.55.15 and KCL 3.5.3.
   
   No documentation or upgrade guide change: a LATEST consumer now routes 
records it skipped before, only after an iterator expired.
   
   Backport: main only; the code builds on CAMEL-25569, which is on main only.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26968](https://issues.apache.org/jira/browse/CAMEL-26968).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built camel-aws2-kinesis and the core modules with the formatter and 
import-sort plugins and ran the camel-aws2-kinesis unit tests. I did not run 
the full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to