Production results of push-based shuffle after rolling out to 100% of Spark workloads at LinkedIn

Hi, We previously raised the SPIP for push-based shuffle in SPARK-30602 . Thanks for the reviews from the community, a significant portion of the code has already been merged. In the meantime, we have been continuing to improve the solution

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

Hi Joseph, Would be interested in discussing your thoughts for how push-based shuffle could help with continuous mode in SS. We have discussed internally at LinkedIn with our Samza peers as well as with Alibaba Flink team for applicability of push-based shuffle on streaming engines, especially

Re: Push-based shuffle SPIP

The linked doc with detailed information of the branch does not seem to be shareable publicly. We have created a copy of the doc which should be publicly accessible. https://docs.google.com/document/d/1Q5m7YAp0HyG_TNFL4p_bjQgzzw33ik5i49Vr86UNZgg/edit?usp=sharing - Min Shen Staff Software

Push-based shuffle SPIP

We raised this SPIP ticket in https://issues.apache.org/jira/browse/SPARK-30602 earlier this year. Since then, we have progressed in multiple fronts, including: * Our work is published in VLDB 2020. The final version of the paper is attached in the SPIP ticket. * We have further enhanced and

Re: Enabling push-based shuffle in Spark

Our paper summarizing this work of push-based shuffle was recently accepted by VLDB 2020. We have uploaded a preprint version of the paper to the JIRA ticket , along with the production results we have so far. - Min Shen Staff Software

Re: Enabling push-based shuffle in Spark

Hi Wenchen, Glad to know that you like this idea. We also looked into making this pluggable in our early design phase. While the ShuffleManager API for pluggable shuffle systems does provide quite some room for customized behaviors for Spark shuffle, we feel that it is still not enough for this

Re: Enabling push-based shuffle in Spark

Hi Reynold, Thanks for the comments. Although in the SPIP doc, a big portion of the problem motivation is around optimizing small random reads for shuffle, I believe the benefit of this design is beyond that. In terms of the approach we take, it is true that the map phase would still need to

Enabling push-based shuffle in Spark

I'd like to start a discussion on enabling push-based shuffle in Spark. This is meant to address issues with existing shuffle inefficiency in a large-scale Spark compute infra deployment. Facebook's previous talks on SOS shuffle and