Hi, Xintong, > I think we can also publish the prototype codes so the community can better understand and help with it.
Ok, I agree on the point. I will prepare and publish the code recently. Rui, > Kindly reminder: the image of CIP-6[1] cannot be loaded. Thanks for the reminder. I've updated the images. Best, Yuxin Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道： > Thanks Yuxin for driving this proposal! > > Kindly reminder: the image of CIP-6[1] cannot be loaded. > > [1] > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > Best, > Rui > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > +1 for this proposal. > > > > We have been prototyping this feature internally at Alibaba for a couple > of > > months. Yuxin, I think we can also publish the prototype codes so the > > community can better understand and help with it. > > > > Best, > > > > Xintong > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <tanyuxinw...@gmail.com> > wrote: > > > > > Hi all, > > > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid > > shuffle > > > integration with > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between > > > memory, disk, and > > > remote storage to improve performance and job stability. Concurrently, > > > Apache Celeborn > > > provides a stable, performant, scalable remote shuffle service. This > > > integration proposal is to > > > harness the benefits from both hybrid shuffle and Celeborn > > simultaneously. > > > > > > Note that this proposal has two parts. > > > 1. The Flink-side modifications are in FLIP-459[1]. > > > 2. The Celeborn-side changes are in CIP-6[2]. > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > > > > > Best, > > > Yuxin > > > > > >
