Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19833#discussion_r153758037
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
---
@@ -106,6 +105,13 @@ class BasicWriteTaskStatsTracker(hadoopConf:
Configuration)
override def getFinalStats(): WriteTaskStats = {
statCurrentFile()
+
+ // Reports bytesWritten and recordsWritten to the Spark output metrics.
+ Option(TaskContext.get()).map(_.taskMetrics().outputMetrics).foreach {
outputMetrics =>
--- End diff --
I don't know if there is any Spark job/stage UI test in the SQL module, and
since this is a pretty trivial change, I just tested it manually. Previously,
`BasicWriteTaskStatsTrackerSuite` failed with NPE because I didn't handle null
TaskContext here. Now I wrap it with `Option` to make it null safe.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
