allthingssecurity opened a new pull request, #29992: URL: https://github.com/apache/camel/pull/29992
# Description [CAMEL-26947](https://issues.apache.org/jira/browse/CAMEL-26947), part of CAMEL-26610. A deterministic simulation test (camel-test-dst) of the plain camel-kafka consumer, in the `dst` package of its tests, in the same form as the SQS and camel-kafka-share worlds: `KafkaConsumerDstTest` (1000 seeds), `KafkaWorld` and `FakeKafka`. Test only; no main code changes. What the world models: the real consumer (`KafkaConsumer`, `KafkaFetchRecords`, the commit managers) and Camel's routing run on the virtual-time loop; only Kafka is faked, as a `Consumer` dynamic proxy returned by a `KafkaClientFactory` set on the component. The fake is written from the `KafkaConsumer` and `Consumer` Javadoc of kafka-clients 4.3, for the classic group protocol with an eager assignor, and quotes the rules it follows: poll with `max.poll.records`, pause and resume, seek, auto-commit while polling, before a revocation and at close, `commitSync` that blocks and can time out (applied or not, chosen by the seed, as the Javadoc does not say), ordered `commitAsync` callbacks from a later call, `committed`, `wakeup`, the rebalance listener (an exception of it is rethrown from poll), commits taken while the group prepares a rebalance, and a phantom member that joins and leaves the group so that partitions are revoked and assigned again while records are processed. kafka-cli ents' `MockConsumer` is not used, as the JIRA allowed: its poll returns at once, its commits never fail or take time and it has no group, so it cannot drive the timing this world is about (the reason is in the `FakeKafka` Javadoc). The seed picks the commit mode (Kafka's auto-commit, manual synchronous or asynchronous commits in the route after every record or only the last one of a poll, or batching with Camel's commits), `breakOnFirstError`, `pollOnError` (ERROR_HANDLER, DISCARD, RETRY, RECONNECT), the poll timeout, `maxPollRecords` and the fault rates, and injects consumer creation failures, poll failures (also in the middle of a batch), failed or slow commits, failed or slow processing, route restarts (also while the consumer polls), suspend and resume with a second request while the consumer still handles the first, and rebalances. Invariants: a committed offset never passes a record the route has not finished with; a suspended route stops fetching; once the faults cleared every record is finished, the committed offsets reach the end of each partition and the route holds one Kafka consumer; the consumer uses the client as its Javadoc requires; no Kafka consumer is left open after the route stopped. Limits, written in the world's Javadoc: `consumersCount` is 1 (each consumer thread loops for ever, and on the one stack of the harness a second looping thread nested above the first would never let it resume); the operator does not stop or suspend the route while an exchange is in the route (the shutdown cannot see it finish on the one stack) or while the consumer waits for the task that creates its Kafka consumer (a stop interrupts that wait only with real threads); a stop while the consumer thread waits in poll is driven. Not explored: batching with Kafka's auto-commit, batching with `breakOnFirstError`, manual commits from another thread, seekTo and offset repositories. `breakOnFirstError` with the synchronous manual commits is not in the default set while it is looked at separately (`-Ddst.kafka.syncBreakOnFirstError=true` adds it). Other changes: - `camel-test-dst` as a test dependency of camel-kafka, and its byte code agent on the surefire command line, as for the other worlds. - The test log of camel-kafka goes to a size-capped rolling file (50 MB, one backup), as a failing world can log a lot. - The `@DstWorld` annotation lists the world in the component JSON (and its catalog copy) and in the table of the camel-test-dst page (and its catalog copy, byte-identical). Tests: - On main: 5000 of 5000 seeds pass (about 9 ms a seed), and 2000 of 2000 batching seeds (`-Ddst.kafka.modes=BATCH`). Seeds 8 and 77 replayed in two JVMs give the same trace hashes. - Self-check against fixed bugs: on the code before CAMEL-25313 (reverted by hand on main, as the old commit no longer applies) 222 of 2000 seeds fail; on the code before CAMEL-25756 272 of 2000 batching seeds fail. Main passes both seed sets. - camel-kafka unit tests in the `org.apache.camel` packages: 1332 run (332, plus 1000 seeds), 0 failures, 0 errors, 1 skipped (the two tests in `org.apache.kafka.clients` were not run). What was not run locally: the ITs (they need Docker) and the full root build. I built and tested offline with the versions cached on my machine: kafka-clients 4.3.1 (main has 4.3.2), commons-lang3 3.21.0 and Guava 33.7.2 (main has 3.22.0 and 33.8.0-jre). # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26947](https://issues.apache.org/jira/browse/CAMEL-26947). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-test-dst, camel-kafka-common and camel-kafka, including the formatter and import-sort plugins. The generated files touched are the component JSON with its catalog copy, and the table of the camel-test-dst page with its catalog copy (byte-identical). I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
