[
https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huangtengfei updated SPARK-36954:
---------------------------------
Description:
[Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L740]
is a function specific for SS.
Now it could be triggered on a batch dataset, and add a specific rule to
eliminate in analyze phase. User can call this API and nothing happens, it may
be a little bit confused.
If the usage is not as expected, maybe we can just fast fail it with explicit
message, and also we do not have to keep on extra rule to do this.
> Fast fail with explicit err msg when calling withWatermark on non-streaming
> dataset
> -----------------------------------------------------------------------------------
>
> Key: SPARK-36954
> URL: https://issues.apache.org/jira/browse/SPARK-36954
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Structured Streaming
> Affects Versions: 3.1.2
> Reporter: huangtengfei
> Priority: Minor
>
> [Dataset.withWatermark|https://github.com/apache/spark/blob/v3.2.0-rc7/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L740]
> is a function specific for SS.
> Now it could be triggered on a batch dataset, and add a specific rule to
> eliminate in analyze phase. User can call this API and nothing happens, it
> may be a little bit confused.
> If the usage is not as expected, maybe we can just fast fail it with explicit
> message, and also we do not have to keep on extra rule to do this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
