Hi Yuxin, As one of the main contributors of hybrid shuffle, I am very happy to see this improvement.
IMO, hybrid shuffle should gradually replace the default batch shuffle implementation, because it is theoretically a superset of blocking shuffle(i.e. SortMergeShuffle / BoundedBlockingShuffle). Today, with the popularity of K8S, more and more people suffer from insufficient disk space problem when running batch jobs on cloud environment. Therefore, supporting elastic remote storage is a crucial part of this process. This FLIP is good for me on the whole, but I still have some problems: 1. How to optimize the broadcast result partition? Is it the same as the original hybrid shuffle that uses the full spilling strategy? IIUC, this FLIP proposes to remove the spilling operation from the memory layer to the disk layer, does it also lose the ability to consume data from memory in this case? This problem can be extended to any multi-consumer situation, such as partition reuse and speculative execution. 2. With regard to the number of reader-side memory management, the original hybrid shuffle did not calculate an accurate backlog for performance reasons, so it is necessary to reserve two exclusive buffers for each channel to avoid some performance deterioration. Can the new proposal completely avoid this problem? At the same time, does N in the formula mean that one exclusive buffer is reserved for each channel? If so, why can't all of them floating? Best regards, Weijie Yuxin Tan <tanyuxinw...@gmail.com> 于2023年3月6日周一 13:51写道： > Hi everyone, > > I would like to start a discussion on FLIP-301: Hybrid Shuffle supports > Remote Storage[1]. > > In the cloud-native environment, it is difficult to determine the > appropriate > disk space for Batch shuffle, which will affect job stability. > > This FLIP is to support Remote Storage for Hybrid Shuffle to improve the > Batch job stability in the cloud-native environment. > > The goals of this FLIP are as follows. > 1. By default, use the local memory and disk to ensure high shuffle > performance if the local storage space is sufficient. > 2. When the local storage space is insufficient, use remote storage as > a supplement to avoid large-scale Batch job failure. > > Looking forward to hearing from you. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage > > Best, > Yuxin >
