allthingssecurity commented on PR #30133:
URL: https://github.com/apache/camel/pull/30133#issuecomment-6910429700

   Thanks for the review, Claus. Pushed 6d6c1aa64b34.
   
   - **[Important] The world no longer excuses this case:** the 
`dst.kinesis.strictLatest` flag is gone. `FakeKinesis` counts the records 
before a LATEST iterator as skipped by design only for the first iterator of a 
shard in a consumer, so a later LATEST iterator after an expiry, which this PR 
replaces with `AT_TIMESTAMP`, would now fail the seed. The `KinesisWorld` 
Javadoc paragraph and the property in the "Running the tests" section of 
test-dst.adoc (and its catalog copy) are updated. The `@DstWorld` annotation 
did not change, so the component JSON stays as it is. With the change, 
`KinesisConsumerDstTest` passes 5000 of 5000 seeds, and the camel-aws2-kinesis 
unit tests pass (5064 run including the 5000 seeds, 0 failures).
   
     One thing the module run showed: three consumer tests 
(`KinesisConsumerClosedShardWithFailTest`, 
`KinesisConsumerClosedShardWithSilentTest`, `KinesisConsumerMultiShardTest`) 
started a consumer and never stopped it. Its poll thread kept running into the 
DST test of the same JVM, and with this PR a poll reads the clock 
(`Instant.now()` for the LATEST start), which the harness reports as a read 
from a thread outside the run: 772 of 5000 seeds failed that way, all at once, 
none on an invariant. Those three tests now stop their consumer in an 
`@AfterEach`. Running `KinesisConsumerDstTest` on its own did not show it, 
which is probably why your 2000 seeds passed.
   - **[Suggestion] Upgrade guide:** added one sentence to the existing 4.24 
entry: with `LATEST`, a shard from which the consumer has not routed a record 
yet continues from the time the consumer started reading that shard.
   - **[Question] Moving `requestedAt` back by about a second:** I left it as 
it is. The time is already taken before the `GetShardIterator` request is sent, 
so it is early by the latency of that call, and with NTP the skew between a 
host and AWS is usually a few milliseconds. A fixed second would route records 
stored before the consumer started, which a `LATEST` user asked to skip, and 
after a route restart (a new consumer, no checkpoint) it could route again 
records the previous consumer already routed. So it trades a rare miss on a 
badly skewed clock for duplicates on every restart of a quiet shard, which does 
not look clearly better to me. Happy to add it, maybe as a small option, if you 
prefer.
   
   _Claude Code on behalf of allthingssecurity_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to