[ https://issues.apache.org/jira/browse/SPARK-35246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333404#comment-17333404 ]
Apache Spark commented on SPARK-35246: -------------------------------------- User 'jose-torres' has created a pull request for this issue: https://github.com/apache/spark/pull/32371 > Streaming-batch intersects are incorrectly allowed through > UnsupportedOperationsChecker > --------------------------------------------------------------------------------------- > > Key: SPARK-35246 > URL: https://issues.apache.org/jira/browse/SPARK-35246 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.0.0, 3.1.0 > Reporter: Jose Torres > Priority: Major > Fix For: 3.2.0 > > > The UnsupportedOperationChecker currently rejects streaming intersects only > if both sides are streaming, but they don't work if even one side is > streaming. The following simple test, for example, fails with a cryptic > None.get error because the state store can't plan itself properly. > {code:java} > test("intersect") { > val input = MemoryStream[Long] > val df = input.toDS().intersect(spark.range(10).as[Long]) > testStream(df) ( > AddData(input, 1L), > CheckAnswer(1) > ) > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org