Github user vackosar commented on a diff in the pull request:
https://github.com/apache/spark/pull/21919#discussion_r208902157
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala ---
@@ -198,11 +198,14 @@ class SourceProgress protected[sql](
* during a trigger. See [[StreamingQueryProgress]] for more information.
*
* @param description Description of the source corresponding to this
status.
+ * @param numOutputRows Number of rows written to the sink or -1 for
Continuous Mode (temporarily)
+ * or Sink V1 (until decommissioned).
* @since 2.1.0
*/
@InterfaceStability.Evolving
class SinkProgress protected[sql](
- val description: String) extends Serializable {
+ val description: String,
+ val numOutputRows: Long) extends Serializable {
--- End diff --
numOutputRows is used here to be similar to SourceProgress#numInputRows. I
agree that I could change from SinkProgress#numOutputRows to
SinkProgress#numInputRows as it may make bit more sense from what actually
happens. Would you and others be in favor of that?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
