GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/18438
[SPARK-18294][CORE] Implement commit protocol to support `mapred` package's
committer
## What changes were proposed in this pull request?
This PR makes the following changes:
- Implement a new commit protocol `HadoopMapRedCommitProtocol` which
support the old `mapred` package's committer;
- Refactor SparkHadoopWriter and SparkHadoopMapReduceWriter, now they are
combined together, thus we can support write through both mapred and mapreduce
API by the new SparkHadoopWriter, a lot of duplicated codes are removed.
After this change, it should be pretty easy for us to support the committer
from both the new and the old hadoop API at high level.
## How was this patch tested?
No major behavior change, passed the existing test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark SparkHadoopWriter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18438.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18438
----
commit 5aa7a2b3cc9dcdc2b4d23644b7d8c025f291dfa3
Author: Xingbo Jiang <[email protected]>
Date: 2017-06-27T15:50:08Z
commit protocol support old hadoop api
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
