junaiddshaukat opened a new issue, #39042:
URL: https://github.com/apache/beam/issues/39042
Tracking issue: #18479
Follows: #38987 (WatermarkManager part 2, merged)
Unblocks: GroupByKey
## Summary
A Kafka Serde for KStreamsPayload so the envelope can cross a topic
boundary. Until now KStreamsPayload only flowed in-JVM via
ProcessorContext#forward, so no serialization was needed; GroupByKey introduces
the first real topic (the key-based repartition topic, per the plan agreed with
@je-ik), which needs the payload serialized. Splitting this out as its own
small PR ahead of GBK.
## Scope
- KStreamsPayloadSerde<T>: parameterized by the Coder<WindowedValue<T>> for
the
data variant (different topics carry different element types; the watermark
variant is coder-independent). Wire format is a one-byte discriminator +
body:
data = [0x00][windowedValueCoder-encoded value]; watermark =
[0x01][long millis][int sourcePartition][int totalSourcePartitions].
- Unit tests: round-trip of data, watermark, and terminal MAX watermark
payloads, plus an unknown-tag failure.
## Out of scope
- Wiring the serde into the GBK repartition / watermark fan-out — that's the
GroupByKey PR.
## Notes
- Assumes non-null payloads: the topics it's used on (repartition, watermark
fan-out) are not log-compacted, so no tombstones occur.
## Testing
./gradlew :runners:kafka-streams:check green; 4 unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
