Hi devs, Xia Sun, Lei Yang, and I would like to initiate a discussion about FLIP-469: Supports Adaptive Optimization of StreamGraph.
This FLIP is the second in the series on adaptive optimization of StreamGraph and follows up on FLIP-468 [1]. As we proposed in FLIP-468 to enable the scheduler to recognize and access the StreamGraph, in this FLIP, we will propose a mechanism for adaptive optimization of StreamGraph. It allows the scheduler to dynamically adjust the logical execution plan at runtime. This mechanism is the base of adaptive optimization strategies, such as adaptive broadcast join and skewed join optimization. Unlike the traditional execution of jobs based on a static StreamGraph, this mechanism will progressively determine StreamGraph during runtime. The determined StreamGraph will be transformed into a specific JobGraph, while the indeterminate part will allow Flink to flexibly adjust according to real-time job status and actual input conditions. Note that this FLIP focuses on the introduction of the mechanism and does not introduce any actual optimization strategies; these will be discussed in subsequent FLIPs. For more details, please refer to FLIP-469 [2]. We look forward to your feedback. Best, Xia Sun, Lei Yang and Junrui Lee [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-469%3A+Supports+Adaptive+Optimization+of+StreamGraph
