Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/15949#discussion_r89237754
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
---
@@ -235,4 +236,49 @@ class StreamingAggregationSuite extends StreamTest
with BeforeAndAfterAll {
CheckLastBatch(("a", 30), ("b", 3), ("c", 1))
)
}
+
+ test("prune results by time, complete mode") {
+ import testImplicits._
+ import StreamingAggregationSuite._
+ clock = new StreamManualClock
+
+ val inputData = MemoryStream[Long]
+
+ val aggregated =
+ inputData.toDF()
+ .groupBy($"value")
+ .agg(count("*"))
+ .as[(Long, Long)]
+ .where('value >= current_timestamp().cast("long") - 10L)
+
+ testStream(aggregated, Complete)(
+ StartStream(ProcessingTime("10 seconds"), triggerClock = clock),
+
+ // advance clock to 10 seconds
+ AddData(inputData, 0L, 5L, 5L, 10L),
+ AdvanceManualClock(10 * 1000),
+ AssertOnQuery { _ => clock.getTimeMillis() === 10 * 1000 },
+ CheckLastBatch((0L, 1), (5L, 2), (10L, 1)),
+
+ // advance clock to 20 seconds
+ AddData(inputData, 15L, 15L, 20L),
+ AdvanceManualClock(10 * 1000),
+ CheckLastBatch((10L, 1), (15L, 2), (20L, 1)),
+
+ // advance clock to 30 seconds
+ AddData(inputData, 0L),
+ AdvanceManualClock(10 * 1000),
+ CheckLastBatch((20L, 1)),
+
+ // advance clock to 40 seconds
+ AddData(inputData, 25L, 30L, 40L, 45L),
+ AdvanceManualClock(10 * 1000),
+ CheckLastBatch((30L, 1), (40L, 1), (45L, 1))
--- End diff --
I dont see how this tests whether the watermark, and timestamp is recovered
correctly from log. nothing is being stopped.
I think there should be another test that actually stops the stream, drops
the complete mode table and restarts it from checkpoint, and see if the table
gets recreated to the same value.
something like
```
val query = inputData.toDF()....
queryName("aggregates").option("checkpointLocation", cpPath).start()
memoryStream.add(....)
query.processAllAvailable()
checkAnswer(table("aggregates"), expectedResults)
query.stop()
spark.drop("aggregates")
clock.add(1000000)
val query2 = inputData.toDF()....
queryName("aggregates").option("checkpointLocation", cpPath).start()
query.processAllAvailable()
checkAnswer(table("aggregates"), expectedResults) // verify that
increasing the clock while query was down does not change result of the last
batch
```
See this for more inspiration -
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala#L472
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
