Hi all, I would like to start a discussion on CIP-6 Support Flink hybrid shuffle integration with Apache Celeborn[1]. Celeborn provides a stable, performant, scalable remote shuffle service. Concurrently, Flink hybrid shuffle supports transitions between memory, disk, and remote storage to improve performance and job stability. This integration proposal is to harness the benefits from both Celeborn and hybrid shuffle simultaneously.
Note that this proposal has two parts. 1. The Celeborn-side changes are in CIP-6[1]. 2. The Flink-side modifications are in FLIP-459[2]. Looking forward to everyone's feedback and suggestions. Thank you! [1] https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn Best, Yuxin
