Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20859#discussion_r175579879
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -160,6 +160,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoDir =>
throwError("InsertIntoDir is not supported with streaming
DataFrames/Datasets")
+ case e: EventTimeWatermark =>
+ val statefulChildren = e.collect {
+ case a: Aggregate if a.isStreaming => a
+ case d: Deduplicate if d.isStreaming => d
+ case f: FlatMapGroupsWithState if f.isStreaming => f
--- End diff --
Should be for joins as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
