bersprockets opened a new pull request, #46190:
URL: https://github.com/apache/spark/pull/46190
This is a backport of #45763 to branch-3.5.
### What changes were proposed in this pull request?
Modify `LateralJoin` to include right-side plan output in `allAttributes`.
### Why are the changes needed?
In the following example, the view v1 is cached, but a query of v1 does not
use the cache:
```
CREATE or REPLACE TEMP VIEW t1(c1, c2) AS VALUES (0, 1), (1, 2);
CREATE or REPLACE TEMP VIEW t2(c1, c2) AS VALUES (0, 1), (1, 2);
create or replace temp view v1 as
select *
from t1
join lateral (
select c1 as a, c2 as b
from t2)
on c1 = a;
cache table v1;
explain select * from v1;
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastHashJoin [c1#180], [a#173], Inner, BuildRight, false
:- LocalTableScan [c1#180, c2#181]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
false] as bigint)),false), [plan_id=113]
+- LocalTableScan [a#173, b#174]
```
The canonicalized version of the `LateralJoin` node is not consistent when
there is a join condition. For example, for the above query, the join condition
is canonicalized as follows:
```
Before canonicalization: Some((c1#174 = a#167))
After canonicalization: Some((none#0 = none#167))
```
You can see that the `exprId` for the second operand of `EqualTo` is not
normalized (it remains 167). That's because the attribute `a` from the
right-side plan is not included `allAttributes`.
This PR adds right-side attributes to `allAttributes` so that references to
right-side attributes in the join condition are normalized during
canonicalization.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New test.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
