Hi everyone, In some scenarios we met a requirement that some operators want to send records to theirs downstream operators with an multicast communication pattern. In detail, for some records, the operators want to send them according to the partitioner (for example, Rebalance), and for some other records, the operators want to send them to all the connected operators and tasks. Such a communication pattern could be viewed as a kind of multicast: it does not broadcast every record, but some record will indeed be sent to multiple downstream operators.
However, we found that this kind of communication pattern seems could not be implemented rightly if the operators have multiple consumers with different parallelism, using the customized partitioner. To solve the above problem, we propose to enhance the support for such kind of irregular communication pattern. We think there may be two options: 1. Support a kind of customized operator events, which share much similarity with Watermark, and these events can be broadcasted to the downstream operators separately. 2. Let the channel selector supports multicast, and also add the separate RecordWriter implementation to avoid impacting the performance of the channel selector that does not need multicast. The problem and options are detailed in https://docs.google.com/document/d/1npi5c_SeP68KuT2lNdKd8G7toGR_lxQCGOnZm_hVMks/edit?usp=sharing We are also wondering if there are other methods to implement this requirement with or without changing Runtime. Very thanks for any feedbacks ! Best, Yun