junaiddshaukat commented on PR #40068: URL: https://github.com/apache/beam/pull/40068#issuecomment-5631019078
> I have one design question - should we just compute target partition and maybe leave the payload empty, because it only signals to the downstream transform it should close bundle? You are right that the consumer needs none of this, and carrying the targets is better. Changed. The marker now carries the target partitions. The producer computes them, which it can do because ShuffleByKeyProcessor already knows its own partition and the upstream count, and the downstream count is known at translation time. That also removes the empty case you spotted: instead of encoding an empty target set, a producer with nothing to address emits no marker at all, so the empty set is now rejected by the factory as a bug rather than represented on the wire. On making it fully empty, I checked and I do not think we can. StreamPartitioner.partitions(topic, key, value, numPartitions) gets the downstream count but no task context, so a partitioner has no way to learn which partition is calling it. Something has to travel in the payload for the routing to be possible at all. Carrying the targets is the smaller of the two, since it is what actually gets used and the rule then lives in one place instead of being re-derived on the other side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
