GitHub user ryux1 added a comment to the discussion: DataFusion-Federation: Union Flattening Across Executors
Yes, the flattening is intentional in core DataFusion, but the loss of this particular pushdown opportunity is a federation-planner limitation rather than a reason that UNION flattening is generally wrong. `OptimizeUnions` explicitly flattens nested `LogicalPlan::Union` inputs. Later, `FederationOptimizerRule` searches for the largest subtree whose scans all resolve to one `FederationProvider`. Once the union is flat, the A/A/B union is ambiguous at the parent and each child is considered separately, so the two A inputs are no longer presented as one subtree to the SQL federation optimizer. This behavior is currently pinned by `datafusion-federation/src/sql/mod.rs::basic_sql_federation_test`: its A/A/B query expects three final SQL statements, including separate statements for `table_a1` and `table_a2`. Disabling `optimize_unions` preserves the accidental binary-tree grouping and therefore enables the A-side pushdown, but it is fragile because grouping then depends on parser plan shape. The durable fix belongs in `datafusion-federation`: when processing a `Union`, group compatible inputs by federation provider / `compute_context`, build a union subtree for each compatible group, federate those groups, and leave a final local union across different executors. That would produce one remote `UNION ALL` for A and one query for B regardless of whether core DataFusion flattened the input first. So: pushing the same-executor union down is better when that executor supports it, but it should be implemented as federation-aware regrouping with capability/error handling, not by globally disabling the core union optimization. GitHub link: https://github.com/apache/datafusion/discussions/22168#discussioncomment-18317590 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
