junaiddshaukat opened a new pull request, #38689:
URL: https://github.com/apache/beam/pull/38689
## Summary
Second sub-issue under the Kafka Streams runner GSoC 2026 project.
Adds URN-dispatch translation framework and the first concrete transform
translator (Impulse), per design doc §4.1.
## What's in this PR
- URN dispatch in `KafkaStreamsPipelineTranslator` walking via
`QueryablePipeline`.
- `ImpulseTranslator` + `ImpulseProcessor` using a dedicated per-application
bootstrap topic (`__beam_impulse_<applicationId>`) and a persistent state
store keyed `fired` to prevent re-emission across restarts.
- `KafkaStreamsTranslationContext` now holds the `Topology` and a
PCollection-id → processor-name map.
- `KafkaStreamsPipelineRunner.run` translates and starts a `KafkaStreams`
application; returns a `KafkaStreamsPortablePipelineResult`.
- `processing.guarantee=exactly_once_v2` set internally (not a user option).
## Tests
- `KafkaStreamsPipelineTranslatorTest` — Impulse success path + unsupported
URN still rejected (uses GroupByKey URN as the negative case).
- `ImpulseTranslatorTest` — TopologyTestDriver behavioural tests:
exactly one empty `byte[]` in `GlobalWindow`, no re-emit on second trigger.
## Validation
- `./gradlew :runners:kafka-streams:check` green locally.
## Deferred / out of scope (documented in code)
- Bootstrap topic auto-creation (`AdminClient`) — design doc §12.1.
Production currently requires the topic to pre-exist; tests bypass via
`TopologyTestDriver`.
- Watermark advancement to `TIMESTAMP_MAX_VALUE` — Kafka Streams has no
native Beam watermark; this moves to the (future) runner-side watermark
manager rather than the record timestamp.
- ExecutableStage / stateless ParDo — next sub-issue.
Closes #38616
Refs #18479
cc @je-ik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
