tdas commented on a change in pull request #28040: [SPARK-31278][SS] Fix
StreamingQuery output rows metric
URL: https://github.com/apache/spark/pull/28040#discussion_r398916867
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationSuite.scala
##########
@@ -280,7 +280,8 @@ class StreamingDeduplicationSuite extends
StateStoreMetricsTest {
{ // State should have been cleaned if flag is set, otherwise should
not have been cleaned
if (flag) assertNumStateRows(total = 1, updated = 1)
else assertNumStateRows(total = 7, updated = 1)
- }
+ },
+ AssertOnQuery(q => q.lastProgress.sink.numOutputRows == 0L)
Review comment:
arent there other tests for the no-data flag in other stateful query suites?
basically, there is no test till now that is testing whether # output rows
generated by no-data-batches are computed correctly. Here is there are no rows
that are output, so cant say whether this 0 is some sort of default 0 or
computed correctly to be 0. And streaming dedup wont be non-zero in
no-data-batches. So maybe try streaming aggs in append mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
