Anish Mahto created SPARK-60525:
-----------------------------------
Summary: Wrap RelationChanges directly in NamedStreamingRelation
for streaming CHANGES queries
Key: SPARK-60525
URL: https://issues.apache.org/jira/browse/SPARK-60525
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Anish Mahto
Fix For: 4.4.0
When `spark.sql.streaming.queryEvolution.enableSourceEvolution=true`, every
streaming source must be named, which SQL users do with `IDENTIFIED BY`. For
streaming `CHANGES` queries, combining `IDENTIFIED BY` with an alias or
watermark fails analysis:
```sql
SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src AS c;
SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src
WATERMARK _commit_timestamp DELAY OF INTERVAL 1 MINUTE;
```
For a `CHANGES` query, `AstBuilder.visitStreamTableName` builds the
`NamedStreamingRelation` over the aliased and watermarked plan, not over
`RelationChanges`. With both clauses, the plan looks roughly like:
```
NamedStreamingRelation(src)
+- EventTimeWatermark
+- SubqueryAlias(c)
+- RelationChanges
```
`RelationChanges` resolves to `SubqueryAlias(cat.t, StreamingRelationV2)`. The
`NameStreamingSources` analysis rule only removes `NamedStreamingRelation` and
passes the name down when its child is a streaming relation, or a single
`SubqueryAlias` directly over one. Here the watermark, the alias, or both sit
in between, so the rule never matches. The `NamedStreamingRelation` is left in
the plan unresolved, and analysis fails even though the statement is valid. For
`SELECT *`, the error is `INVALID_USAGE_OF_STAR_OR_REGEX`.
The fix is to build the `NamedStreamingRelation` directly over
`RelationChanges`, which is what the non-`CHANGES` branch and
`DataStreamReader.changes()` already do:
```
EventTimeWatermark
+- SubqueryAlias(c)
+- NamedStreamingRelation(src)
+- RelationChanges
```
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
