GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/10080
[SPARK-11328] [SQL] Improve error message when hitting this issue
The issue is that the output commiter is not idempotent and retry attempts
will
fail because the output file already exists. It is not safe to clean up the
file
as this output committer is by design not retryable. Currently, the job
fails
with a confusing file exists error. This patch is a stop gap to tell the
user
to look at the top of the error log for the proper message.
This is difficult to test locally as Spark is hardcoded not to retry.
Manually
verified by upping the retry attempts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nongli/spark spark-11328
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10080.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10080
----
commit 78a2172e4261b52297c66fbe5624b9f04258c2d4
Author: Nong Li <[email protected]>
Date: 2015-10-27T06:43:56Z
[SPARK-11382] [SQL] Improve error message when hitting this issue.
The issue is that the output commiter is not idempotent and retry attempts
will
fail because the output file already exists. It is not safe to clean up the
file
as this output committer is by design not retryable. Currently, the job
fails
with a confusing file exists error. This patch is a stop gap to tell the
user
to look at the top of the error log for the proper message.
This is difficult to test locally as Spark is hardcoded not to retry.
Manually
verified by upping the retry attempts.
commit 0d35d604bdd7291cacda7e893a266387fe93b5df
Author: Nong Li <[email protected]>
Date: 2015-11-30T20:52:00Z
Code review.
commit 3ed9f67d4d9ebc1b7061357347dc717433acad33
Author: Nong Li <[email protected]>
Date: 2015-11-30T21:58:27Z
CR
commit c4375ec7396bf914b0b74b03c40e774ff4bc9477
Author: Nong Li <[email protected]>
Date: 2015-12-01T19:19:52Z
Add another place to catch the exception.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
