AnishMahto opened a new pull request, #59761: URL: https://github.com/apache/spark/pull/59761
### What changes were proposed in this pull request? When `spark.sql.streaming.queryEvolution.enableSourceEvolution=true`, every streaming source must be named, which SQL users do with `IDENTIFIED BY`. For streaming `CHANGES` queries, combining `IDENTIFIED BY` with an alias or watermark fails analysis: ```sql SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src AS c; SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src WATERMARK _commit_timestamp DELAY OF INTERVAL 1 MINUTE; ``` For a `CHANGES` query, `AstBuilder.visitStreamTableName` builds the `NamedStreamingRelation` over the aliased and watermarked plan, not over `RelationChanges`. With both clauses, the plan looks roughly like: ``` NamedStreamingRelation(src) +- EventTimeWatermark +- SubqueryAlias(c) +- RelationChanges ``` `RelationChanges` resolves to `SubqueryAlias(cat.t, StreamingRelationV2)`. The `NameStreamingSources` analysis rule only removes `NamedStreamingRelation` and passes the name down when its child is a streaming relation, or a single `SubqueryAlias` directly over one. Here the watermark, the alias, or both sit in between, so the rule never matches. The `NamedStreamingRelation` is left in the plan unresolved, and analysis fails even though the statement is valid. For `SELECT *`, the error is `INVALID_USAGE_OF_STAR_OR_REGEX`. The fix is to build the `NamedStreamingRelation` directly over `RelationChanges`, which is what the non-`CHANGES` branch and `DataStreamReader.changes()` already do: ``` EventTimeWatermark +- SubqueryAlias(c) +- NamedStreamingRelation(src) +- RelationChanges ``` ### Why are the changes needed? Bug fix; today users cannot specify `IDENTIFIED BY` along with an alias or watermark for a SQL `STREAM` statement, even though both should be supported. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Opus 5.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
