Hi devs, I re-sorted out and supplemented the 'FLIP-339[1] Support Adaptive Partition Selection for StreamPartitioner' based on Flink JIRA[2].
Flink offers multiple partition strategies, some of which bind data to downstream subtasks, while others do not (e.g., shuffle, rescale, rebalance). For [Data not bound to subtasks] scenarios, overloaded sub-task-nodes may slow down the processing of Flink jobs, leading to backpressure and data lag. Dynamically adjusting the partition of data to subtasks based on the processing load of downstream operators helps achieve a peak-shaving and valley-filling effect, thereby striving to maintain the throughput of Flink jobs. The raw discussions could be found in the Flink JIRA[2]. I really appreciate developers involved in the discussion for the valuable help and suggestions in advance. Please refer to the FLIP[1] wiki for more details about the proposed design and implementation. Welcome any feedback and opinions on this proposal. [1] https://cwiki.apache.org/confluence/x/nYyzDw [2] https://issues.apache.org/jira/browse/FLINK-31655 Best regards, Yuepeng Pan
