Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20386#discussion_r164649253
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala
---
@@ -148,7 +148,8 @@ private[continuous] class EpochCoordinator(
logDebug(s"Epoch $epoch has received commits from all partitions.
Committing globally.")
// Sequencing is important here. We must commit to the writer before
recording the commit
// in the query, or we will end up dropping the commit if we restart
in the middle.
- writer.commit(epoch, thisEpochCommits.toArray)
+ thisEpochCommits.foreach(writer.add(_))
--- End diff --
is it possible to call `add` once the commit message arrives?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
