Hi, Han.
The configuration for External Shuffle Service(ESS) in YARN has to be
configured in yarn-site.xml for NodeManagers, as it is an auxiliary service
in NodeManager. We will try to improve the documentation for enabling push
based shuffle. Thanks for the feedback.
For the straggler issue, is
Hi Ye,
This is super super helpful! It wasn't obvious to me from the documentation
that this property needed to be set in the yarn-site.xml file, as all other
configurations in the main spark configuration page are set through spark
conf. It was particularly confusing because this property, like m
Hi, Han.
Thanks for trying out the push based shuffle.
Please make sure you configure both the Spark client side configuration and
server side configurations.
The client side configuration looks good, and from the error message, looks
like you are missing the server side configurations.
Please refe
+CC zhouye...@gmail.com
On Mon, May 23, 2022 at 7:11 AM Han Altae-Tran wrote:
> Hi,
>
> First of all, I am very thankful for all of the amazing work that goes
> into this project! It has opened up so many doors for me! I am a long
> time Spark user, and was very excited to start working with th
Hi,
First of all, I am very thankful for all of the amazing work that goes into
this project! It has opened up so many doors for me! I am a long time Spark
user, and was very excited to start working with the push-based shuffle
service for an academic paper we are working on, but I encountered som
