junaiddshaukat opened a new pull request, #39770: URL: https://github.com/apache/beam/pull/39770
## Summary Merges the Kafka Streams runner from its feature branch (`feat/18479-kafka-streams-runner-skeleton`) into `master`. Tracking issue: #18479. Developed over this summer as a Google Summer of Code project, mentored by @je-ik. Opened as **[WIP]** while the lazy consensus thread on dev@ runs. It is not to be merged before that thread closes. ## What this is A portable runner that translates a Beam pipeline into a Kafka Streams topology and runs user code over the Fn API. What makes it different from the other runners is that Kafka Streams is a library rather than a cluster. There is no job manager and no resource manager to operate: a pipeline is an ordinary JVM process reading from and writing to Kafka, scaled by starting more copies of that process. State, fault tolerance and exactly-once come from Kafka itself, through consumer groups, changelog topics and transactions. ## It is not in the standard build The runner's subprojects are only included when `-Pwith-kafka-streams-runner` is passed: ``` ./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:build ``` Without the flag they are not in the build at all, so nothing reaches anyone who has not asked for it and no release artifact contains it. `beam_PreCommit_Java_Kafka_Streams_Runner` passes the flag, so the runner is still built and tested on every PR that touches it and cannot rot unnoticed. The point is to give the runner somewhere it can be built, used and worked on, rather than have it quietly die on a branch. If it becomes stable enough the flag comes off; if it does not, it can be dropped without affecting anyone, because no release ever shipped it. ## State Supported, and covered by tests: bounded and unbounded reads, stateless `ParDo` including multiple outputs, `GroupByKey` and `Combine`, global, fixed and sliding windows with the default trigger and allowed lateness, `Flatten`, `Redistribute`, metrics, and exactly-once via Kafka transactions. Known gaps, each tracked: side inputs (#39628), stateful `ParDo` and user timers (#39629), merging windows and custom `WindowFn`s (#39630), splittable `DoFn` (#39631), `TestStream` (#39632), reading a source in parallel (#39626). Known bugs rather than missing features: bundles are not closed after a bounded time (#39633) — `maxBundleTimeMs` is accepted and has no effect, because closing a bundle from a wall-clock punctuator duplicated output against a real broker and the cause is not yet understood. There may be others not yet found. ## Testing ``` ./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:build # 105 unit tests ./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:validatesRunner # 59 tests ``` Both pass against current `master`. Beam's Python portable suite also runs against the runner, exercising it from a non-Java SDK; it needs a broker, so it is not part of any aggregate build. Checked that the default build is untouched: `./gradlew projects` lists no Kafka Streams projects, and `javaPreCommit` configures with none of its tasks. ## Changes to the branch as merged The feature branch's own CI workflow (`beam_KafkaStreamsRunner_FeatureBranch.yml`) is removed here, since it only triggered on `feat/18479-*` and has no purpose on `master`. `beam_PreCommit_Java_Kafka_Streams_Runner` is the workflow that remains. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
