[ANNOUNCE] Apache Spark 3.0.3 released

2021-06-24 Thread Yi Wu
We are happy to announce the availability of Spark 3.0.3! Spark 3.0.3 is a maintenance release containing stability fixes. This release is based on the branch-3.0 maintenance branch of Spark. We strongly recommend all 3.0 users to upgrade to this stable release. To download Spark 3.0.3, head

Re: [DISCUSS] Spark cannot identify the problem executor

2020-09-13 Thread Yi Wu
: > Thanks Yi Wu and Sean. Here I mean shuffle data, and without shuffle > service. > > spark.blacklist.application.fetchFailure.enabled=true; seems to be the > answer and I was not noticed about it, thanks for pointing it out. And I > will give a try. > > However I doubt how it

Re: [DISCUSS] Spark cannot identify the problem executor

2020-09-11 Thread Yi Wu
What do you mean by "read from executor A"? I can think of several paths for an executor to read something from another remote executor: 1. shuffle data If the executor fails to fetch the shuffle data, I think it will result in the FetchFiled for the task. For this case, blacklist can identify