GitHub user themodernlife opened a pull request:
https://github.com/apache/spark/pull/2450
[SPARK-3595] Respect configured OutputCommitters when calling
saveAsHadoopFile
Addresses the issue in https://issues.apache.org/jira/browse/SPARK-3595,
namely saveAsHadoopFile hardcoding the OutputCommitter. This is not ideal when
running Spark jobs that write to S3, especially when running them from an EMR
cluster where the default OutputCommitter is a DirectOutputCommitter.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/themodernlife/spark spark-3595
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2450.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2450
----
commit 8b6be94801ca33bca32aa574b1a8f6a76760869d
Author: Ian Hummel <[email protected]>
Date: 2014-09-15T15:38:59Z
Add ability to specify OutputCommitter, espcially useful when writing to an
S3 bucket from an EMR cluster
commit 4359664b1d557d55b0579023df809542386d5b8c
Author: Ian Hummel <[email protected]>
Date: 2014-09-18T20:18:57Z
Add an example showing usage
commit a11d9f3806e6a8d06d13417af9f27bfd3795334b
Author: Ian Hummel <[email protected]>
Date: 2014-09-18T20:52:17Z
Fix formatting
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
