Repository: spark
Updated Branches:
refs/heads/master a99d284c1 -> 9b6f24202
[MINOR][CORE] Log committer class used by HadoopMapRedCommitProtocol
## What changes were proposed in this pull request?
When HadoopMapRedCommitProtocol is used (e.g., when using saveAsTextFile() or
saveAsHadoopFile() with RDDs), it's not easy to determine which output committer
class was used, so this PR simply logs the class that was used, similarly to
what
is done in SQLHadoopMapReduceCommitProtocol.
## How was this patch tested?
Built Spark then manually inspected logging when calling saveAsTextFile():
```scala
scala> sc.setLogLevel("INFO")
scala> sc.textFile("README.md").saveAsTextFile("/tmp/out")
...
18/05/29 10:06:20 INFO HadoopMapRedCommitProtocol: Using output committer class
org.apache.hadoop.mapred.FileOutputCommitter
```
Author: Jonathan Kelly <[email protected]>
Closes #21452 from ejono/master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9b6f2420
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9b6f2420
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9b6f2420
Branch: refs/heads/master
Commit: 9b6f24202f6f8d9d76bbe53f379743318acb19f9
Parents: a99d284
Author: Jonathan Kelly <[email protected]>
Authored: Mon Jun 11 16:41:15 2018 -0500
Committer: Sean Owen <[email protected]>
Committed: Mon Jun 11 16:41:15 2018 -0500
----------------------------------------------------------------------
.../apache/spark/internal/io/HadoopMapRedCommitProtocol.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9b6f2420/core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala
----------------------------------------------------------------------
diff --git
a/core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala
b/core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala
index ddbd624..af0aa41 100644
---
a/core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala
+++
b/core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala
@@ -31,6 +31,8 @@ class HadoopMapRedCommitProtocol(jobId: String, path: String)
override def setupCommitter(context: NewTaskAttemptContext): OutputCommitter
= {
val config = context.getConfiguration.asInstanceOf[JobConf]
- config.getOutputCommitter
+ val committer = config.getOutputCommitter
+ logInfo(s"Using output committer class
${committer.getClass.getCanonicalName}")
+ committer
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
