GitHub user ejono opened a pull request:
https://github.com/apache/spark/pull/21452
[MINOR][CORE] Log committer class used by HadoopMapRedCommitProtocol
## What changes were proposed in this pull request?
When HadoopMapRedCommitProtocol is used (e.g., when using saveAsTextFile()
or
saveAsHadoopFile() with RDDs), it's not easy to determine which output
committer
class was used, so this PR simply logs the class that was used, similarly
to what
is done in SQLHadoopMapReduceCommitProtocol.
## How was this patch tested?
Built Spark then manually inspected logging when calling saveAsTextFile():
```scala
scala> sc.setLogLevel("INFO")
scala> sc.textFile("README.md").saveAsTextFile("/tmp/out")
...
18/05/29 10:06:20 INFO HadoopMapRedCommitProtocol: Using output committer
class org.apache.hadoop.mapred.FileOutputCommitter
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ejono/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21452.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21452
----
commit 9881d9c6a2b1d56e69bb06ee27fd8706f6e0fe43
Author: Jonathan Kelly <jonathak@...>
Date: 2018-05-29T16:36:02Z
[MINOR][CORE] Log committer class used by HadoopMapRedCommitProtocol
When HadoopMapRedCommitProtocol is used (e.g., when using saveAsTextFile()
or
saveAsHadoopFile() with RDDs), it's not easy to determine which output
committer
class was used, so this PR simply logs the class that was used, similarly
to what
is done in SQLHadoopMapReduceCommitProtocol.
Built Spark then manually inspected logging when calling saveAsTextFile():
```scala
scala> sc.setLogLevel("INFO")
scala> sc.textFile("README.md").saveAsTextFile("/tmp/out")
...
18/05/29 10:06:20 INFO HadoopMapRedCommitProtocol: Using output committer
class org.apache.hadoop.mapred.FileOutputCommitter
```
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
