GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/15861
[SPARK-18294][CORE] Implement commit protocol to support `mapred` package's
committer
## What changes were proposed in this pull request?
This PR makes the following changes:
1. Implement a new commit protocol named `HadoopMapRedCommitProtocol` which
support the old `mapred` package's committer;
2. Refactor `SparkHadoopWriter` and `SparkHadoopMapReduceWriter`, now they
are combined together, thus we can support write through both `mapred` and
`mapreduce` API by the new `SparkHadoopWriter`, a lot of duplicated codes are
removed;
3. Moved `SparkHadoopWriterUtils` to a seprated file.
## How was this patch tested?
This PR is not changing any behavior, so it is tested by the existing test
cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark mapred-commit-protocol
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15861.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15861
----
commit 75ae7e27dbbc3e8c39cdb73388e314df0f348292
Author: jiangxingbo <[email protected]>
Date: 2016-11-10T10:03:15Z
commit protocol support mapred API
commit 2483570e4917dc265aeeeaa71ac6f6836c83c203
Author: jiangxingbo <[email protected]>
Date: 2016-11-12T06:23:17Z
refactor.
commit 7f7303e0f605539710ed321b8e61c847a2d8efd4
Author: jiangxingbo <[email protected]>
Date: 2016-11-12T07:37:23Z
clean code.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
