HeartSaVioR commented on a change in pull request #24936: [SPARK-24634][SS] Add
a new metric regarding number of rows later than watermark plus allowed delay
URL: https://github.com/apache/spark/pull/24936#discussion_r329964118
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -795,7 +795,7 @@ class FlatMapGroupsWithStateSuite extends
StateStoreMetricsTest {
}
},
CheckNewAnswer(("c", "-1")),
- assertNumStateRows(total = 0, updated = 0)
+ assertNumStateRows(total = 0, updated = 1, lateInput = 0)
Review comment:
> The "updated" value is override with 0 when exposing if the batch was run
with empty data
This can be considered as a bug instead, as we have actual case where batch
with empty data has nonzero "updated" value. I guess it's added when batch with
empty data wasn't introduced, and missed to address the case. I'll deal with
this as well. (The fix is actually in this PR, but given the issue is
orthogonal to the origin issue of this PR, so better to have new PR.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
