allthingssecurity opened a new pull request, #29992:
URL: https://github.com/apache/camel/pull/29992

   # Description
   
   [CAMEL-26947](https://issues.apache.org/jira/browse/CAMEL-26947), part of 
CAMEL-26610.
   
   A deterministic simulation test (camel-test-dst) of the plain camel-kafka 
consumer, in the `dst` package of its tests, in the same form as the SQS and 
camel-kafka-share worlds: `KafkaConsumerDstTest` (1000 seeds), `KafkaWorld` and 
`FakeKafka`. Test only; no main code changes.
   
   What the world models: the real consumer (`KafkaConsumer`, 
`KafkaFetchRecords`, the commit managers) and Camel's routing run on the 
virtual-time loop; only Kafka is faked, as a `Consumer` dynamic proxy returned 
by a `KafkaClientFactory` set on the component. The fake is written from the 
`KafkaConsumer` and `Consumer` Javadoc of kafka-clients 4.3, for the classic 
group protocol with an eager assignor, and quotes the rules it follows: poll 
with `max.poll.records`, pause and resume, seek, auto-commit while polling, 
before a revocation and at close, `commitSync` that blocks and can time out 
(applied or not, chosen by the seed, as the Javadoc does not say), ordered 
`commitAsync` callbacks from a later call, `committed`, `wakeup`, the rebalance 
listener (an exception of it is rethrown from poll), commits taken while the 
group prepares a rebalance, and a phantom member that joins and leaves the 
group so that partitions are revoked and assigned again while records are 
processed. kafka-cli
 ents' `MockConsumer` is not used, as the JIRA allowed: its poll returns at 
once, its commits never fail or take time and it has no group, so it cannot 
drive the timing this world is about (the reason is in the `FakeKafka` Javadoc).
   
   The seed picks the commit mode (Kafka's auto-commit, manual synchronous or 
asynchronous commits in the route after every record or only the last one of a 
poll, or batching with Camel's commits), `breakOnFirstError`, `pollOnError` 
(ERROR_HANDLER, DISCARD, RETRY, RECONNECT), the poll timeout, `maxPollRecords` 
and the fault rates, and injects consumer creation failures, poll failures 
(also in the middle of a batch), failed or slow commits, failed or slow 
processing, route restarts (also while the consumer polls), suspend and resume 
with a second request while the consumer still handles the first, and 
rebalances.
   
   Invariants: a committed offset never passes a record the route has not 
finished with; a suspended route stops fetching; once the faults cleared every 
record is finished, the committed offsets reach the end of each partition and 
the route holds one Kafka consumer; the consumer uses the client as its Javadoc 
requires; no Kafka consumer is left open after the route stopped.
   
   Limits, written in the world's Javadoc: `consumersCount` is 1 (each consumer 
thread loops for ever, and on the one stack of the harness a second looping 
thread nested above the first would never let it resume); the operator does not 
stop or suspend the route while an exchange is in the route (the shutdown 
cannot see it finish on the one stack) or while the consumer waits for the task 
that creates its Kafka consumer (a stop interrupts that wait only with real 
threads); a stop while the consumer thread waits in poll is driven. Not 
explored: batching with Kafka's auto-commit, batching with `breakOnFirstError`, 
manual commits from another thread, seekTo and offset repositories. 
`breakOnFirstError` with the synchronous manual commits is not in the default 
set while it is looked at separately (`-Ddst.kafka.syncBreakOnFirstError=true` 
adds it).
   
   Other changes:
   - `camel-test-dst` as a test dependency of camel-kafka, and its byte code 
agent on the surefire command line, as for the other worlds.
   - The test log of camel-kafka goes to a size-capped rolling file (50 MB, one 
backup), as a failing world can log a lot.
   - The `@DstWorld` annotation lists the world in the component JSON (and its 
catalog copy) and in the table of the camel-test-dst page (and its catalog 
copy, byte-identical).
   
   Tests:
   - On main: 5000 of 5000 seeds pass (about 9 ms a seed), and 2000 of 2000 
batching seeds (`-Ddst.kafka.modes=BATCH`). Seeds 8 and 77 replayed in two JVMs 
give the same trace hashes.
   - Self-check against fixed bugs: on the code before CAMEL-25313 (reverted by 
hand on main, as the old commit no longer applies) 222 of 2000 seeds fail; on 
the code before CAMEL-25756 272 of 2000 batching seeds fail. Main passes both 
seed sets.
   - camel-kafka unit tests in the `org.apache.camel` packages: 1332 run (332, 
plus 1000 seeds), 0 failures, 0 errors, 1 skipped (the two tests in 
`org.apache.kafka.clients` were not run).
   
   What was not run locally: the ITs (they need Docker) and the full root 
build. I built and tested offline with the versions cached on my machine: 
kafka-clients 4.3.1 (main has 4.3.2), commons-lang3 3.21.0 and Guava 33.7.2 
(main has 3.22.0 and 33.8.0-jre).
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26947](https://issues.apache.org/jira/browse/CAMEL-26947).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built and tested camel-test-dst, camel-kafka-common and camel-kafka, 
including the formatter and import-sort plugins. The generated files touched 
are the component JSON with its catalog copy, and the table of the 
camel-test-dst page with its catalog copy (byte-identical). I did not run the 
full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to