marikomedlock opened a new pull request, #27525:
URL: https://github.com/apache/datafusion/pull/27525
## Which issue does this PR close?
- Closes #27523.
## Rationale for this change
The Substrait consumer turns an N-input `SetRel` union into a left-deep
chain of N−1 two-input `Union` nodes. `OptimizeUnions` later flattens the chain
back into one `Union`, and type coercion is re-run at every level of the chain,
so logical planning grows quadratically with the number of union inputs.
On a plan with 7 unions of up to 186 inputs, building the unions flat cut
`SessionState::optimize` from 3.27s to 2.34s on a 4-vCPU Graviton2, and from
0.535s to 0.332s on an Apple M-series laptop. Numbers are in #27523.
## What changes are included in this PR?
- `union_rels` consumes all inputs and builds one `Union` with
`Union::try_new_with_loose_types`, so the result schema is derived the same way
as before. `UNION DISTINCT` wraps it in one `Distinct`.
- A single input (the secondary side of a 2-input `INTERSECT`) is returned
as-is, as before.
## What is the testing strategy for this PR?
- New tests, each failing on `main`:
- `consume_nary_union_as_single_union`: consumes a 3-input `UNION ALL`
without optimizing and asserts one `Union` with 3 inputs (`main` gives two
nested 2-input unions).
- `consume_nary_union_distinct_as_single_union`: same for a 3-input `UNION
DISTINCT`, asserting one `Distinct` over one 3-input `Union` (`main` gives two
nested `Distinct`s). Uses a new fixture, `union_distinct_nary.substrait.json`.
- `nary_union_distinct_consume`: the optimized plan for the 3-input `UNION
DISTINCT` matches the plan for the equivalent SQL (passes on `main` too,
showing optimized plans are unchanged).
- The existing tests pass unchanged (`cargo test -p datafusion-substrait`),
as do the sqllogictest Substrait round-trip CI steps. `union.slt` in round-trip
mode has the same 2 pre-existing failures on `main` and on this branch
(`NATURAL JOIN`, `FixedSizeList` casts).
## Are there any user-facing changes?
The logical plan produced by the Substrait consumer for a union of more than
2 inputs is now one N-ary `Union` instead of nested 2-input unions. After
optimization the plans are the same as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]