Github user vackosar commented on a diff in the pull request:
https://github.com/apache/spark/pull/21919#discussion_r208901343
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -58,6 +61,7 @@ case class WriteToDataSourceV2Exec(writer:
DataSourceWriter, query: SparkPlan) e
val useCommitCoordinator = writer.useCommitCoordinator
val rdd = query.execute()
val messages = new Array[WriterCommitMessage](rdd.partitions.length)
+ val totalNumRowsAccumulator = new LongAccumulator()
--- End diff --
I am using here threadsafe java native LongAccumulator. Spark cluster
accumulator is not needed here and would be also a bit out of the ordinary in
the code base area. Note that StreamingQueryProgress is reported as an event
object. That is why I am not using the cluster accumulator here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
