huangtengfei created SPARK-36954:
------------------------------------
Summary: Fast fail with explicit err msg when calling
withWatermark on non-streaming dataset
Key: SPARK-36954
URL: https://issues.apache.org/jira/browse/SPARK-36954
Project: Spark
Issue Type: Improvement
Components: SQL, Structured Streaming
Affects Versions: 3.1.2
Environment:
[Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L740]
is a function specific for SS.
Now it could be triggered on a batch dataset, and add a specific rule to
eliminate in analyze phase. User can call this API and nothing happens, it may
be a little bit confused.
If the usage is not as expected, maybe we can just fast fail it with explicit
message, and also we do not have to keep on extra rule to do this.
Reporter: huangtengfei
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
