marikomedlock opened a new pull request, #27525:
URL: https://github.com/apache/datafusion/pull/27525

   ## Which issue does this PR close?
   
   - Closes #27523.
   
   ## Rationale for this change
   
   The Substrait consumer turns an N-input `SetRel` union into a left-deep 
chain of N−1 two-input `Union` nodes. `OptimizeUnions` later flattens the chain 
back into one `Union`, and type coercion is re-run at every level of the chain, 
so logical planning grows quadratically with the number of union inputs.
   
   On a plan with 7 unions of up to 186 inputs, building the unions flat cut 
`SessionState::optimize` from 3.27s to 2.34s on a 4-vCPU Graviton2, and from 
0.535s to 0.332s on an Apple M-series laptop. Numbers are in #27523.
   
   ## What changes are included in this PR?
   
   - `union_rels` consumes all inputs and builds one `Union` with 
`Union::try_new_with_loose_types`, so the result schema is derived the same way 
as before. `UNION DISTINCT` wraps it in one `Distinct`.
   - A single input (the secondary side of a 2-input `INTERSECT`) is returned 
as-is, as before.
   
   ## What is the testing strategy for this PR?
   
   - New tests, each failing on `main`:
     - `consume_nary_union_as_single_union`: consumes a 3-input `UNION ALL` 
without optimizing and asserts one `Union` with 3 inputs (`main` gives two 
nested 2-input unions).
     - `consume_nary_union_distinct_as_single_union`: same for a 3-input `UNION 
DISTINCT`, asserting one `Distinct` over one 3-input `Union` (`main` gives two 
nested `Distinct`s). Uses a new fixture, `union_distinct_nary.substrait.json`.
   - `nary_union_distinct_consume`: the optimized plan for the 3-input `UNION 
DISTINCT` matches the plan for the equivalent SQL (passes on `main` too, 
showing optimized plans are unchanged).
   - The existing tests pass unchanged (`cargo test -p datafusion-substrait`), 
as do the sqllogictest Substrait round-trip CI steps. `union.slt` in round-trip 
mode has the same 2 pre-existing failures on `main` and on this branch 
(`NATURAL JOIN`, `FixedSizeList` casts).
   
   ## Are there any user-facing changes?
   
   The logical plan produced by the Substrait consumer for a union of more than 
2 inputs is now one N-ary `Union` instead of nested 2-input unions. After 
optimization the plans are the same as before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to