LukaZdravic opened a new pull request, #59322: URL: https://github.com/apache/spark/pull/59322
### What changes were proposed in this pull request? `AsOfJoin` now keeps its direction (backward, forward, or nearest), and `SortMergeAsOfJoinExec` reads it from the plan. Before, the exec guessed it from the shape of the as-of condition. The DataFrame path stores the direction it is given. The SQL path takes it from the `MATCH_CONDITION` operator after the left table's operand is put first. ### Why are the changes needed? The guess only works when the condition has the left side first. Any other shape runs as nearest: ``` l.ts >= r.ts -> backward r.ts <= l.ts -> nearest (same condition, wrong scan) ``` No rule writes the condition that way today, but any rewrite that did would silently change the scan. A backward join run as nearest can then pick a different row when right rows share an as-of value. The plan already knows the direction, so it should pass it on. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests in `SortMergeAsOfJoinSuite` and `AsOfJoinSortMergeSQLSuite`. The first uses a test-only rule that puts the right side first, and checks that the join keeps its direction. The second checks that a right-operand-first `MATCH_CONDITION` gets the flipped direction under both analyzers. I also tightened the nearest and strict `>`/`<` tests, so a wrong direction now picks a different row. With the old guess put back, the new `SortMergeAsOfJoinSuite` test fails. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
