junaiddshaukat commented on code in PR #40068:
URL: https://github.com/apache/beam/pull/40068#discussion_r3986879588
##########
runners/kafka-streams/proto/src/main/proto/kafka_streams_payload.proto:
##########
@@ -50,9 +50,32 @@ message KafkaStreamsPayload {
bytes value = 1;
}
- // Exactly one variant is set; the oneof case discriminates data vs
watermark.
+ // A request to close the open bundle and flush its output.
+ //
+ // A bundle has to be bounded in time as well as in size, or on a sparse
stream the elements
+ // already fed to it wait for the next watermark. Closing it from a
wall-clock punctuator does
+ // not work: transactions are committed by the Kafka Streams runtime in the
background, are
+ // agnostic to punctuations, and are deliberately not exposed, so a bundle
cannot be aligned with
+ // one. Instead a source emits this marker on its own punctuator and it
travels the topology as
+ // an ordinary record, so a stage closes its bundle from process() rather
than beside it.
+ //
+ // The partition fields are what let the marker be targeted rather than
broadcast. Broadcasting
+ // would deliver one flush per upstream partition, so a downstream partition
would see N times
+ // more flushes than the configured interval. Instead the producing
partition addresses a slice
+ // of the downstream partitions, and the slices tile the whole range, so
each downstream
+ // partition receives exactly one flush per interval.
Review Comment:
Made it concise, thanks for pushing on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
