Dandandan opened a new pull request, #24517:
URL: https://github.com/apache/datafusion/pull/24517
## Which issue does this PR close?
- Closes #.
## Rationale for this change
`SortMergeJoinExec` emits every column of both inputs. `HashJoinExec` can
emit a
subset, so `ProjectionPushdown` absorbs a projection into it; for a sort
merge join
the projection has to stay as its own operator above the join.
Projection pushdown does try to push into the join's children, but only when
each
side's columns stay together in the output, and it gives up entirely when a
join key
is not part of the output. Joining on an expression is the common case:
```sql
select * from t1 right join t2 on t1.c3 = t2.c3 -- c3 needs a cast
```
```text
ProjectionExec: expr=[c1@0 as c1, ..., c4@8 as c4]
SortMergeJoinExec: join_type=Right, on=[(CAST(t1.c3 AS Decimal128(10,
2))@4, c3@2)]
```
The cast column is only there to be joined on, and every operator above the
join
carries it until the projection removes it.
## What changes are included in this PR?
`SortMergeJoinExec` takes an optional projection, set with
`with_projection`, the same
shape as `HashJoinExec`'s. It is applied to the join's output, reported in
`EXPLAIN` as
`projection=[..]`, carried through plan properties, statistics and
serialization, and
kept when children are replaced.
`try_swapping_with_projection` embeds the projection into the join when it
cannot push
it into the children, so the query above becomes:
```text
SortMergeJoinExec: join_type=Right, on=[(CAST(t1.c3 AS Decimal128(10, 2))@4,
c3@2)], projection=[c1@0, c2@1, c3@2, c4@3, c1@5, c2@6, c3@7, c4@8]
```
## Are these changes tested?
Yes:
- a projection pushdown test for a projection that interleaves the two
sides, which
the existing pushdown cannot handle
- a serialization round trip
- existing sqllogictests, whose plans lose a `ProjectionExec` in four places
TPC-H SF10 and TPC-DS SF1 with `prefer_hash_join = false` are unchanged in
runtime
(within noise, medians of alternating runs). This removes an operator and
narrows the
join's output; it does not yet stop the join from building the columns it
drops.
## Are there any user-facing changes?
`EXPLAIN` shows `projection=[..]` on a sort merge join that has one, and one
fewer
`ProjectionExec` above it. `SortMergeJoinExec` gains `with_projection`,
`contains_projection` and a public `projection` field; nothing existing
changes shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
