allthingssecurity opened a new pull request, #28068: URL: https://github.com/apache/camel/pull/28068
# Description [CAMEL-25741](https://issues.apache.org/jira/browse/CAMEL-25741) `ShardIteratorHandler.getShardIterators()` decided which shards the `aws2-ddbstream` consumer reads from two signals, and neither means that a shard was read: 1. A tracked shard with children in the shard tree was replaced by its children ("Inactive shard, move down to its children"). DynamoDB creates the children of a shard when it closes it, while the closed shard still holds the records the consumer has not read; the DynamoDB Streams guide says "an application must always process a parent shard before it processes a child shard". With `streamIteratorType=FROM_START` the closed root shards were left after the first `GetRecords` response; on a table with several partitions, every shard that had closed meanwhile was left when the tree was refreshed for another one. Their remaining records were never routed. 2. An empty map of shard iterators re-initialised the consumer from `streamIteratorType`. The map is empty when every shard being read was read to its end: a table with one partition at each shard rotation (a shard lives up to four hours, according to the AWS Database Blog), or several shards that end in the same poll. `FROM_START` then routed records of the stream again from its roots (at every poll, without ever reading the child shards, when the closed roots were read to their end in one response); `FROM_LATEST` took `LATEST` iterators of the child shards and skipped the records already written to them. CAMEL-24154 described this single-partition case; its fix (`pendingClosedShards`) only runs while another shard is still tracked. This change uses the configured iterator type once, for the start position (and again only while no shard was found at all), and leaves a shard only when `GetRecords` returned no next shard iterator. Its children are then read from `TRIM_HORIZON`, as before. A shard read to its end whose children `DescribeStream` does not list yet is looked up again at the next poll; before, it was dropped and the empty map re-initialised the consumer. The class is the only production change. Cost: while a shard that was read to its end has no listed children, each poll makes one `DescribeStream` call (paginated), as main does at every poll once the map is empty. A stream that is disabled (its last shards close without children) keeps the consumer calling `DescribeStream` at each poll. What changes for users (4.24 upgrade guide entry under a new `camel-aws2-ddb` heading): a closed shard is read to its end before its children, so no records are skipped, and the consumer no longer routes the stream again after a rotation with `FROM_START`. Not changed: - a shard whose iterator expired before any record was routed still restarts at `TRIM_HORIZON` (CAMEL-24154); - children are still read while other shards are being read; records of one partition key are in order only within a shard; - a closed shard is followed as soon as `DescribeStream` lists one of its children: a second child listed later is not picked up (the caveat noted in CAMEL-16594). Tests: - New `Ddb2StreamConsumerShardRotationTest` (5 tests): a real route and an in-memory `DynamoDbStreamsClient` with closed and open shards. A closed root with 5 records and `maxResultsPerRequest=2` (`FROM_START`); a closed root and its child (`FROM_START`); the child of the only shard, with a record written to it before the consumer reads the end of the parent (`FROM_LATEST`); two shards that rotate together, one with more records than a response returns (`FROM_LATEST`); a child that `DescribeStream` lists only after the consumer read the end of its parent (`FROM_LATEST`). Each test waits until the expected records are routed, then for ten more `GetRecords` calls, and checks that each record was routed once. - Without the change (main's `ShardIteratorHandler`, same tests) all five end with an error (Awaitility `ConditionTimeoutException` around the assertion): `routed so far: [1, 2, 6] ==> expected: <6> but was: <3>`, `routed so far: [1, 2, 1, 2, 1, 2, ...]` (the stream read again at every poll, the child's record never), `routed so far: [2] ==> expected: <2> but was: <1>`, `routed so far: [1, 2, 8, 9] ==> expected: <9> but was: <4>`, `routed so far: [2] ==> expected: <3> but was: <1>`. - `ShardIteratorHandlerTest.shouldProgressThroughTreeWhenShardIteratorsAreRetrievedRepeatedly` expected the move to the children at each call without any record read. It now checks that the root is kept until it is read to its end, then reads each level to its end before the next call. - With the change, camel-aws2-ddb tests pass: 82 run, 0 failures. The integration tests need AWS and were not run. How it was found: deterministic simulation testing with a fake DynamoDB Streams world written from the DynamoDB Streams API reference (shard history, writes, rotations and splits, faults on every call; the seed also picked `streamIteratorType`, `maxResultsPerRequest` from 1 to 1000, the poll delay and `greedy`). The invariants "every record the consumer is expected to read is routed" and "no record is routed twice" failed for both signals on main; with this change the same seeds pass. The simulation harness is not part of this PR. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-aws2-ddb, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
