HeartSaVioR commented on a change in pull request #24936: [SPARK-24634][SS] Add
a new metric regarding number of rows later than watermark plus allowed delay
URL: https://github.com/apache/spark/pull/24936#discussion_r329955888
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -795,7 +795,7 @@ class FlatMapGroupsWithStateSuite extends
StateStoreMetricsTest {
}
},
CheckNewAnswer(("c", "-1")),
- assertNumStateRows(total = 0, updated = 0)
+ assertNumStateRows(total = 0, updated = 1, lateInput = 0)
Review comment:
Maybe that is intention to increase the value, but the "updated" value is
override with 0 when exposing if the batch was run with empty data (that's why
the value was "0" though the row was removed). So the value is applied
conditionally which is confusing. I'd rather say we need to add "removed"
metrics or just follow the approach other state operators are doing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
