LukaZdravic opened a new pull request, #58823:
URL: https://github.com/apache/spark/pull/58823
### What changes were proposed in this pull request?
A scalar ASOF JOIN `MATCH_CONDITION` operand pair is now coerced to the
common type the comparison operator would use, and the comparison, the ordering
distance, and both per-side sort keys are all built from the coerced operands
so they stay consistent. The common type follows the active analyzer mode (ANSI
or default), the same way `>=` coerces the pair. `STRUCT` and `ARRAY` operands
keep the stricter widening rule, because their whole-value sort key cannot
carry a per-field cast.
### Why are the changes needed?
`MATCH_CONDITION (l.d >= r.s)` with a `DATE` (or `TIMESTAMP`) column and a
`STRING` column failed with `ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE`, although
the same pair is a valid `>=` comparison that coerces the string:
```sql
-- Failed before this PR:
SELECT * FROM (VALUES (DATE '2024-01-03')) l(d)
ASOF JOIN (VALUES ('2024-01-01')) r(s)
MATCH_CONDITION (l.d >= r.s);
-- Works (the same comparison):
SELECT DATE '2024-01-03' >= '2024-01-01'; -- true
```
Only removing the type guard is not enough. ASOF JOIN is a sort-merge join:
it sorts the right buffer by its sort key and relies on the `MATCH_CONDITION`
comparison being monotone along that order. The comparison is coerced by the
analyzer (string to date), but the sort keys were built from the raw operands,
so the right side would be sorted as a `STRING` while compared as a `DATE`. A
string sorts lexicographically and a date sorts by value, so the two orders
differ and the join returned the wrong as-of match. Coercing the operands up
front keeps the sort key and the comparison on one type. The same mechanism
also fixes this latent sort bug for string vs number operands.
### Does this PR introduce _any_ user-facing change?
Yes. A `MATCH_CONDITION` that compares a string to a `DATE`, `TIMESTAMP`, or
number is now accepted and coerced, matching the comparison operator. It
previously failed with `ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE`. This
reproduces with ANSI mode on and off.
### How was this patch tested?
- `AsOfJoinMatchConditionTypesSuite`: a scalar string vs temporal/number
pair is compatible and reports the coerced common type; `STRUCT` fields and
`ARRAY` elements still reject a string vs temporal pair.
- `AsOfJoinSQLSuite`: `DATE` vs `STRING` resolves with `DATE` sort keys and
returns the correct row, under ANSI on and off; an incompatible pair
(`TIMESTAMP` vs `DECIMAL`) still errors.
- `join-asof-datatypes.sql` golden: `DATE`, `TIMESTAMP`, and `INT` vs
`STRING` cases. The `INT` vs `STRING` case returns the value-ordered match, not
the lexicographic one, which pins the sort-order fix.
- `join-asof-errors.sql` golden: `TIMESTAMP` vs `DECIMAL` still reports
`ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
