marikomedlock opened a new issue, #27523:
URL: https://github.com/apache/datafusion/issues/27523

   ### Is your feature request related to a problem or challenge?
   
   When the Substrait consumer reads a `SetRel` with N inputs (`UNION ALL` / 
`UNION DISTINCT`), `union_rels` in 
`datafusion/substrait/src/logical_plan/consumer/rel/set_rel.rs` folds them into 
a left-deep chain of N−1 two-input `Union` nodes:
   
   ```
   Union
     Union
       Union
         t1
         t2
       t3
     t4
   ```
   
   `OptimizeUnions` then flattens the chain back into one N-input `Union`, and 
type coercion is re-run at every level of the chain. On wide unions this makes 
logical planning grow quadratically with the number of inputs, before the 
optimizer ends up with the flat union it could have started from.
   
   We hit this with generated queries that `UNION ALL` many tables. One query 
has 7 unions with 186, 186, 186, 91, 91, 91 and 3 inputs. Time spent in 
`SessionState::optimize` (analyzer + logical optimizer) for that plan:
   
   | | 4-vCPU Graviton2 (shared) | Apple M-series laptop |
   |---|---|---|
   | current (nested unions) | 3.27s | 0.535s |
   | flat unions (this change) | 2.34s (−28%) | 0.332s (−38%) |
   | flat unions + skipping expression-free nodes in `SimplifyExpressions` / 
`RewriteSetComparison` (separate PR) | 1.34s (−59%) | 0.199s (−63%) |
   
   Each number is the mean of 9–12 warm runs.
   
   ### Describe the solution you'd like
   
   Have `union_rels` consume all inputs and build a single 
`Union::try_new_with_loose_types` over them, wrapping it in one `Distinct` for 
`UNION DISTINCT`. The resulting schema is the same: `Union` derives its schema 
the same way for N inputs as for 2.
   
   ### Describe alternatives you've considered
   
   Leaving it to `OptimizeUnions`. That works, but the flattening and repeated 
coercion are the cost.
   
   ### Additional context
   
   Related: #17261 (slow planning for unions with many columns), #26602 
(Substrait JOIN/UNION planning).
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to