Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20445#discussion_r166720161
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala
---
@@ -46,49 +46,34 @@ class ForeachSinkSuite extends StreamTest with
SharedSQLContext with BeforeAndAf
.foreach(new TestForeachWriter())
.start()
- // -- batch 0 ---------------------------------------
- input.addData(1, 2, 3, 4)
- query.processAllAvailable()
+ def verifyOutput(expectedVersion: Int, expectedData: Seq[Int]): Unit
= {
+ import ForeachSinkSuite._
- var expectedEventsForPartition0 = Seq(
- ForeachSinkSuite.Open(partition = 0, version = 0),
- ForeachSinkSuite.Process(value = 2),
- ForeachSinkSuite.Process(value = 3),
- ForeachSinkSuite.Close(None)
- )
- var expectedEventsForPartition1 = Seq(
- ForeachSinkSuite.Open(partition = 1, version = 0),
- ForeachSinkSuite.Process(value = 1),
- ForeachSinkSuite.Process(value = 4),
- ForeachSinkSuite.Close(None)
- )
+ val events = ForeachSinkSuite.allEvents()
--- End diff --
This test assumed that the output would arrive in specific order after
repartitioning, which isnt guaranteed. So I rewrote the test to verify the
output in an order-independent way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
