uros-b commented on code in PR #59591:
URL: https://github.com/apache/spark/pull/59591#discussion_r4316108244
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AsOfJoinResolver.scala:
##########
@@ -140,6 +141,22 @@ class AsOfJoinResolver(
}
}
+ /**
+ * Merges the ID mappings of both sides. It also keeps the IDs of hidden
columns like `d.k`:
+ * `MATCH_CONDITION` is resolved again later, and a copy of the same table
on the other side
+ * must not remap them.
+ */
+ private def mergeChildMappings(leftNameScope: NameScope, rightNameScope:
NameScope): Unit = {
Review Comment:
This only protects a hidden column's ID when the ASOF JOIN merges its own
two children. JoinResolver still does:
```scala
//
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala:66-68
expressionIdAssigner.createMappingFromChildMappings(
newOutputIds = leftNameScope.getOutputIds ++ rightNameScope.getOutputIds
)
```
If a copy of the same table meets the hidden column in a regular JOIN below
the ASOF, that child's collected map is already K → K_y. Adding K to
newOutputIds here only prefers identity when K → K is still present as a
competing mapping, so materializedMatchConditionToJoin (336-342) rewrites d.k
to y.k.
```
SELECT d.k, z.k
FROM (
(using_right r LEFT JOIN using_keys d USING (k))
JOIN using_keys y ON y.k = 1
)
ASOF JOIN (VALUES (1), (2)) AS z(k)
MATCH_CONDITION (d.k >= z.k);
```
y.k = 1 keeps one y row, so the left of the ASOF is (d.k=1), (d.k=2),
(d.k=NULL), each with y.k=1. Fixed-point MATCH uses d.k and returns (1, 1), (2,
2). Single-pass MATCH uses y.k, so every row matches z.k=1, including the NULL
d.k row: (1, 1), (2, 1), (NULL, 1). Dual-run reports a plan mismatch. Same hole
if that nested JOIN is the right input of the ASOF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
