Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16098#discussion_r90660126
--- Diff:
core/src/main/scala/org/apache/spark/internal/io/SparkHadoopMapReduceWriter.scala
---
@@ -146,29 +146,32 @@ object SparkHadoopMapReduceWriter extends Logging {
case c: Configurable => c.setConf(hadoopConf)
case _ => ()
}
- val writer = taskFormat.getRecordWriter(taskContext)
- .asInstanceOf[RecordWriter[K, V]]
- require(writer != null, "Unable to obtain RecordWriter")
var recordsWritten = 0L
// Write all rows in RDD partition.
try {
val ret = Utils.tryWithSafeFinallyAndFailureCallbacks {
- while (iterator.hasNext) {
- val pair = iterator.next()
- writer.write(pair._1, pair._2)
-
- // Update bytes written metric every few records
- SparkHadoopWriterUtils.maybeUpdateOutputMetrics(
- outputMetricsAndBytesWrittenCallback, recordsWritten)
- recordsWritten += 1
+ val writer = taskFormat.getRecordWriter(taskContext)
+ require(writer != null, "Unable to obtain RecordWriter")
--- End diff --
Ah, actually, I left a comment about this and then removed because I
thought I might be too much worried.
I at least checked `TextOutputFormat. getRecordWriter(..)` is able to
create the attempt folder and `FileOutputCommitter.abortTask(..)` removes this.
If this is located in the original place, it'd not remove the directory
(although this case would be really rare).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
