Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20859#discussion_r175580404
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -160,6 +160,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoDir =>
throwError("InsertIntoDir is not supported with streaming
DataFrames/Datasets")
+ case e: EventTimeWatermark =>
+ val statefulChildren = e.collect {
+ case a: Aggregate if a.isStreaming => a
+ case d: Deduplicate if d.isStreaming => d
+ case f: FlatMapGroupsWithState if f.isStreaming => f
+ }
+ statefulChildren.foreach { statefulNode =>
+ if (statefulNode.collectFirst{ case e: EventTimeWatermark => e
}.isDefined) {
+ throwError("Watermarks both before and after a stateful
operator in a streaming " +
--- End diff --
This gives the impression that it makes sense but we dont support it. In
fact, its just ill-defined. May change this to something like ... Multiple
watermarks before and after stateful operators is not well-defined in a
streaming query.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
