[
https://issues.apache.org/jira/browse/SPARK-60525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60525:
-----------------------------------
Labels: pull-request-available (was: )
> Wrap RelationChanges directly in NamedStreamingRelation for streaming CHANGES
> queries
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60525
> URL: https://issues.apache.org/jira/browse/SPARK-60525
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anish Mahto
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> When `spark.sql.streaming.queryEvolution.enableSourceEvolution=true`, every
> streaming source must be named, which SQL users do with `IDENTIFIED BY`. For
> streaming `CHANGES` queries, combining `IDENTIFIED BY` with an alias or
> watermark fails analysis:
> ```sql
> SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src AS c;
> SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src
> WATERMARK _commit_timestamp DELAY OF INTERVAL 1 MINUTE;
> ```
> For a `CHANGES` query, `AstBuilder.visitStreamTableName` builds the
> `NamedStreamingRelation` over the aliased and watermarked plan, not over
> `RelationChanges`. With both clauses, the plan looks roughly like:
> ```
> NamedStreamingRelation(src)
> +- EventTimeWatermark
> +- SubqueryAlias(c)
> +- RelationChanges
> ```
> `RelationChanges` resolves to `SubqueryAlias(cat.t, StreamingRelationV2)`.
> The `NameStreamingSources` analysis rule only removes
> `NamedStreamingRelation` and passes the name down when its child is a
> streaming relation, or a single `SubqueryAlias` directly over one. Here the
> watermark, the alias, or both sit in between, so the rule never matches. The
> `NamedStreamingRelation` is left in the plan unresolved, and analysis fails
> even though the statement is valid. For `SELECT *`, the error is
> `INVALID_USAGE_OF_STAR_OR_REGEX`.
> The fix is to build the `NamedStreamingRelation` directly over
> `RelationChanges`, which is what the non-`CHANGES` branch and
> `DataStreamReader.changes()` already do:
> ```
> EventTimeWatermark
> +- SubqueryAlias(c)
> +- NamedStreamingRelation(src)
> +- RelationChanges
> ```
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
