Hi all, I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle integration with Apache Celeborn[1]. Flink hybrid shuffle supports transitions between memory, disk, and remote storage to improve performance and job stability. Concurrently, Apache Celeborn provides a stable, performant, scalable remote shuffle service. This integration proposal is to harness the benefits from both hybrid shuffle and Celeborn simultaneously.
Note that this proposal has two parts. 1. The Flink-side modifications are in FLIP-459[1]. 2. The Celeborn-side changes are in CIP-6[2]. Looking forward to everyone's feedback and suggestions. Thank you! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn [2] https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn Best, Yuxin
