Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20828#discussion_r179635730
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -53,32 +53,24 @@ class ContinuousSuiteBase extends StreamTest {
// A continuous trigger that will only fire the initial time for the
duration of a test.
// This allows clean testing with manual epoch advancement.
protected val longContinuousTrigger = Trigger.Continuous("1 hour")
+
+ override protected implicit val defaultTrigger = Trigger.Continuous(100)
+ override protected val defaultUseV2Sink = true
}
class ContinuousSuite extends ContinuousSuiteBase {
import testImplicits._
- test("basic rate source") {
- val df = spark.readStream
- .format("rate")
- .option("numPartitions", "5")
- .option("rowsPerSecond", "5")
- .load()
- .select('value)
+ test("basic") {
+ val input = MemoryStream[Int]
--- End diff --
What I eventually want to be able to do is to run all our existing test
suites in continuous mode, to ensure that continuous processing can do all the
same things as microbatch execution.
If you're not convinced, we should take this change out of the PR and punt
on it. But I think we will eventually need something much like this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
