GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/9942
[SPARK-11382] [SQL] Improve error message when hitting this issue.
The issue is that the output commiter is not idempotent and retry attempts
will
fail because the output file already exists. It is not safe to clean up the
file
as this output committer is by design not retryable. Currently, the job
fails
with a confusing file exists error. This patch is a stop gap to tell the
user
to look at the top of the error log for the proper message.
This is difficult to test locally as Spark is hardcoded not to retry.
Manually
verified by upping the retry attempts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nongli/spark spark-11328-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9942.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9942
----
commit d19b5c33beaca367bb333b043016b8a9dd26d2e3
Author: Nong Li <[email protected]>
Date: 2015-10-27T06:43:56Z
[SPARK-11382] [SQL] Improve error message when hitting this issue.
The issue is that the output commiter is not idempotent and retry attempts
will
fail because the output file already exists. It is not safe to clean up the
file
as this output committer is by design not retryable. Currently, the job
fails
with a confusing file exists error. This patch is a stop gap to tell the
user
to look at the top of the error log for the proper message.
This is difficult to test locally as Spark is hardcoded not to retry.
Manually
verified by upping the retry attempts.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
