Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19833#discussion_r153687993
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
---
@@ -106,6 +105,13 @@ class BasicWriteTaskStatsTracker(hadoopConf:
Configuration)
override def getFinalStats(): WriteTaskStats = {
statCurrentFile()
+
+ // Reports bytesWritten and recordsWritten to the Spark output metrics.
+ Option(TaskContext.get()).map(_.taskMetrics().outputMetrics).foreach {
outputMetrics =>
--- End diff --
Which test suite covers this newly added logic?
Previously, BasicWriteTaskStatsTrackerSuite seems to fail (8 of 10 test
cases) because TaskContext is absent in the test suite.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
