allthingssecurity opened a new pull request, #29993: URL: https://github.com/apache/camel/pull/29993
# Description [CAMEL-26948](https://issues.apache.org/jira/browse/CAMEL-26948), part of CAMEL-26610. A deterministic simulation test (camel-test-dst) of the camel-aws2-kinesis consumer, in the `dst` package of its tests, in the same form as the SQS world: `KinesisConsumerDstTest` (1000 seeds), `KinesisWorld` and `FakeKinesis`. Test only; no main code changes. What the world models: the real consumer (`Kinesis2Consumer`, its shard monitor and the resume strategy) and Camel's routing run on the virtual-time loop; only Kinesis is faked, as a `KinesisClient` bound as `amazonKinesisClient`. The fake is written from the API reference of GetShardIterator, GetRecords, ListShards, SplitShard and MergeShards and cites it: shard iterators are positions valid for five minutes (`ExpiredIteratorException` after that), the iterator types, answers with fewer records than the limit, a closed shard whose last GetRecords returns no next iterator and lists its child shards, and paginated ListShards. The seed picks the iterator type (TRIM_HORIZON, LATEST), whether the route checkpoints with the resumable EIP and `KinesisResumeStrategy` (the resume cache outlives the consumer, so a restarted route resumes from it), the number of shards, `maxResultsPerRequest`, the poll delay, the shard monitor interval and `shardClosed` (ignore, silent), and injects failed GetRecords, GetShardIterator and ListShards calls, slow answers, slow and failed processing (now and then longer than an iterator is valid), route restarts, suspensions of up to eight minutes (the cached iterators expire) and splits and merges of shards. Invariants: a consumer routes each record at most once (apart from a failed record after its checkpoint, which the resume strategy reads again); a restarted route resumes after its checkpoint; once the faults cleared every record is routed and the checkpoint of each open shard reaches its last record; the consumer uses the API as documented. A record stored before the consumer started a shard of its first shard list with LATEST, and without a checkpoint, is skipped by design. With LATEST, when the iterator of a shard from which the consumer has not routed a record yet expires, the consumer asks for LATEST again, which skips the records stored meanwhile. The default set accepts this only right after such an expiry; `-Ddst.kinesis.strictLatest=true` counts those records as missing, and then 44 of 2000 seeds on main fail. I have not opened a JIRA for it; I can do that if reviewers think the consumer should behave otherwise. Limits, written in the world's Javadoc: the JIRA item "a stop while a poll runs" is not driven. On the one stack of the harness a graceful shutdown cannot see a poll below it finish, so the operator suspends the consumer first (as the shutdown strategy does) and stops or suspends the route only between polls; that item of CAMEL-26948 stays open. Not explored: the KCL consumer, the asynchronous client, a configured shardId, and AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER and AT_TIMESTAMP as the configured iterator type. Not modelled: retention, hash key ranges, enhanced fan-out, read limits beyond the injected throttling, the expiry of a NextToken, shards that close between two pages of a listing, and a stream deleted and created again. Other changes: - `camel-test-dst` as a test dependency of camel-aws2-kinesis, and its byte code agent on the surefire command line, as for the other worlds. - The test log of camel-aws2-kinesis goes to a size-capped rolling file (50 MB, one backup), as a failing world can log a lot. - The `@DstWorld` annotation lists the world in the component JSON (and its catalog copy) and in the table of the camel-test-dst page (and its catalog copy, byte-identical). Tests: - On main: 5000 of 5000 seeds pass (about 9 ms a seed). Seeds 8 and 77 replayed in two JVMs give the same trace hashes. - Self-check against fixed bugs: on the code before CAMEL-25569 (043ac9ec84ab reverted) 305 of 2000 seeds fail (expired iterators and child shards read with LATEST); on the code before CAMEL-25942 (847ee9affb4c reverted) 710 of 2000 seeds fail, all of them seeds with checkpoints. CAMEL-25558, in the JIRA's list of candidates, is a camel-aws2-s3 fix and is not used. - The Kinesis unit tests of the module (`org.apache.camel.component.aws2.kinesis`): 1063 run (63, plus 1000 seeds), 0 failures, 0 errors, 0 skipped. The Firehose tests of the same module were not run. What was not run locally: the ITs (they need Docker) and the full root build. I built and tested offline with the versions cached on my machine: AWS SDK 2.55.15 and the Kinesis client library 3.5.3 (main has 2.57.11 and 3.5.4), commons-lang3 3.21.0 and Guava 33.7.2 (main has 3.22.0 and 33.8.0-jre). # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26948](https://issues.apache.org/jira/browse/CAMEL-26948). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-test-dst, camel-aws-common and camel-aws2-kinesis, including the formatter and import-sort plugins. The generated files touched are the component JSON with its catalog copy, and the table of the camel-test-dst page with its catalog copy, which is byte-identical to the page. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
