allthingssecurity opened a new pull request, #30133: URL: https://github.com/apache/camel/pull/30133
# Description [CAMEL-26968](https://issues.apache.org/jira/browse/CAMEL-26968) Follow-up of #27753 (CAMEL-25569), for the case it left open. Found by the Kinesis DST world of #29993 (CAMEL-26948): its default set accepts this case, and `-Ddst.kinesis.strictLatest=true` fails 44 of 2000 seeds. Reproduced here with a plain unit test, outside the DST harness. Since CAMEL-25569, when a cached shard iterator has expired, `Kinesis2Consumer.getShardIterator` asks for `AFTER_SEQUENCE_NUMBER` with the last record routed from the shard. A shard from which no record was routed yet has no such sequence number, so the consumer asked for the configured `iteratorType` again. With `LATEST` that is a new start point, and the records stored between the first iterator and the new one were skipped. An iterator expires when it is not used for 5 minutes (a route suspended for longer, a long `delay` or back-off, or a poll that waits for a slow route on another shard), and a shard that gets few records is the typical case, as its iterators keep returning nothing. `iteratorType=LATEST` is where the consumer starts a shard ("Start reading just after the most recent record in the shard"); once started, an expired iterator should not move that start, as it does not once a record was routed. Change, in `Kinesis2Consumer.getShardIterator`: - when a request for a new iterator ends up as `LATEST` (after the resume strategy had its say), the consumer remembers the time just before the request, once per shard; - when the iterator of such a shard expires before a record of it was routed, the consumer asks for `AT_TIMESTAMP` with that time ("Start reading from the position denoted by a specific time stamp"). Nothing was routed from the shard, so nothing is routed twice; the records stored since the start are routed. A resume strategy can still change the request, as before; - the other cases are unchanged: after a routed record `AFTER_SEQUENCE_NUMBER`, a child shard of a resharding `TRIM_HORIZON`, the other iterator types the configured one (asking for it again does not move their start). The time comes from the clock of the consumer and Kinesis compares it with the arrival time it gave each record: with a clock ahead of AWS by s seconds the records of those s seconds can still be missed, and with one behind it a few records stored just before the start can be routed. Both are much smaller than the gap a new LATEST iterator skipped. Test, new `expiredIteratorOfAShardWithoutRoutedRecordsDoesNotSkipTheRecordsOfTheGapWithLatest` in `KinesisConsumerPositionTest` (its in-memory stream now gives each record an arrival timestamp and answers `AT_TIMESTAMP`): LATEST, the first poll gets nothing, a is stored, the iterators expire, b is stored; the next poll must route a and b, with an `AT_TIMESTAMP` request; after a routed record the next expiry uses `AFTER_SEQUENCE_NUMBER` again. Without the change the new test fails (the poll routes nothing); with it the camel-aws2-kinesis unit tests pass (64 run, 0 failures). With the change merged locally into the world of #29993, `-Ddst.kinesis.strictLatest=true` passes 2000 of 2000 seeds (44 fail without it), and the default set passes 2000 of 2000. The ITs need LocalStack and were not run locally (CI runs them). The local offline build used the cached AWS SDK 2.55.15 and KCL 3.5.3. No documentation or upgrade guide change: a LATEST consumer now routes records it skipped before, only after an iterator expired. Backport: main only; the code builds on CAMEL-25569, which is on main only. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26968](https://issues.apache.org/jira/browse/CAMEL-26968). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-aws2-kinesis and the core modules with the formatter and import-sort plugins and ran the camel-aws2-kinesis unit tests. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
