srielau opened a new pull request, #57264: URL: https://github.com/apache/spark/pull/57264
### What changes were proposed in this pull request? Second stacked PR for [SPARK-58092](https://issues.apache.org/jira/browse/SPARK-58092) / [SPARK-58123](https://issues.apache.org/jira/browse/SPARK-58123). **Depends on #57251** (parser). Until that lands, the diff vs `master` includes parser commits; the analysis-only delta is `SPARK-58122..SPARK-58123` (1 commit). After #57251 merges, this branch will be rebased onto `master` for a clean diff. **Analysis rule** - Add `ResolveAsOfJoin` to materialize `MATCH_CONDITION` into `asOfCondition`, `orderExpression`, and per-side sort expressions - Expand `USING (...)` into equi-join predicates via `NaturalAndUsingJoinResolution` - Validate MATCH_CONDITION operands: cross-side references, determinism, and type compatibility (including struct/array) **Logical plan** - Extend `AsOfJoin` with `leftSortExprs`, `rightSortExprs`, and `requiresSortMergeAsOfJoin` - Add `resolveMatchComparison` / `matchSortExpressions` helpers shared with the DataFrame API path - Teach `Analyzer` to resolve `AsOfMatchCondition` operands before materialization **Gating** - `CheckAnalysis` rejects SQL ASOF plans when `spark.sql.join.sortMergeAsOfJoin.enabled=false` (`AS_OF_JOIN.SORT_MERGE_REQUIRED`) **Out of scope (follow-up PRs)** - Sort-merge physical operator wiring for multi-column sort keys - `AsOfJoinSortMergeSQLSuite` execution / success-path integration tests ### Why are the changes needed? Parser-only (#57251) produces unresolved `AsOfJoin` nodes with a parsed `AsOfMatchCondition`. Analysis must resolve operands, validate semantics, and produce the same logical shape the existing DataFrame `joinAsOf` path uses before physical planning can run. ### Does this PR introduce _any_ user-facing change? Yes (behind `spark.sql.join.asofJoin.enabled`, default off). With the parser conf enabled, SQL ASOF JOIN now progresses through analysis: valid queries become analyzable logical plans; invalid MATCH_CONDITION expressions fail with structured analysis errors instead of reaching execution unresolved. ### How was this patch tested? Added analysis negative tests to `AsOfJoinSQLSuite`: - `AS_OF_JOIN.SORT_MERGE_REQUIRED` - `ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE` (cross-side and invalid refs) - `ASOF_JOIN_MATCH_CONDITION_INVALID_EXPRESSION` - `ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE` Locally: ``` build/sbt "sql/testOnly org.apache.spark.sql.AsOfJoinSQLSuite" \ "catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.RewriteAsOfJoinSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
