Jose Torres created SPARK-35246: ----------------------------------- Summary: Disable intersects for all streaming queries Key: SPARK-35246 URL: https://issues.apache.org/jira/browse/SPARK-35246 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.1.0, 3.0.0 Reporter: Jose Torres Fix For: 3.2.0
The UnsupportedOperationChecker currently rejects streaming intersects only if both sides are streaming, but they don't work if even one side is streaming. The following simple test, for example, fails with a cryptic None.get error because the state store can't plan itself properly. {code:java} test("intersect") { val input = MemoryStream[Long] val df = input.toDS().intersect(spark.range(10).as[Long]) testStream(df) ( AddData(input, 1L), CheckAnswer(1) ) } {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org