kumarUjjawal opened a new pull request, #26790: URL: https://github.com/apache/datafusion/pull/26790
## Which issue does this PR close? Related to #17261. Complements the fast path for equal schemas added in #24389. ## Rationale for this change Constructing a union with unequal input schemas can take quadratic time in the number of children. The fallback merges every input's field metadata before checking whether that candidate field is nullable. With non-nullable fields, the same full merge is repeated for every child. Select the first nullable field (or the first field when none are nullable) before merging metadata. This makes the fallback linear in child count for a fixed number of fields and metadata entries per input. The shared helper also serves `InterleaveExec`. ## What changes are included in this PR? - Merge metadata once per output field, preserving the selected field, leftmost name, and existing metadata conflict precedence. - Add a test that constructs and executes unions with non-nullable inputs, first/middle/last nullable inputs, conflicting and distinct metadata keys, and checks output rows. - Extend the existing `union_schema` benchmark with non-nullable schemas at 10, 100, and 1,000 children. Its previous cases use nullable fields, so they stop at the first candidate and miss the quadratic behavior. ## What is the testing strategy for this PR? The new regression test passes on both upstream and patched code. All 23 union/interleave tests pass. The full extended workspace test command from `AGENTS.md` and `./dev/rust_lint.sh` passed, including both Clippy configurations with warnings denied. Benchmarks use upstream `c18e5d974` plus the benchmark extension, Rust 1.99.1, `release-nonlto`, and an Apple M5 Pro. The table averages two interleaved runs per binary, with 20 Criterion samples per run. Schemas have 10 fields and two metadata entries per field. | Non-nullable schema case | Children | Upstream | Patched | Speedup | | --- | ---: | ---: | ---: | ---: | | Last child's metadata differs | 10 | 141 µs | 62 µs | 2.3× | | Last child's metadata differs | 100 | 9.24 ms | 0.66 ms | 14.1× | | Last child's metadata differs | 1,000 | 877 ms | 7.40 ms | 118.6× | | Field names differ | 10 | 140 µs | 60 µs | 2.3× | | Field names differ | 100 | 9.38 ms | 0.66 ms | 14.3× | | Field names differ | 1,000 | 866 ms | 7.36 ms | 117.6× | The controls with nullable fields or equal schemas at 4,000 children varied from -0.8% to +6.3% on this shared machine. Background builds affected some measurements, so the larger SQL cases with sorted unions were also checked in a tighter alternating sequence, with four runs per binary: | SQL planning case | Upstream | Patched | Change | | --- | ---: | ---: | ---: | | 50 columns, Int64 | 38.61 ms | 38.40 ms | -0.5% | | 50 columns, UInt64 | 139.48 ms | 139.63 ms | +0.1% | These results establish the constructor improvement independently of #17261; they do not establish an improvement in the SQL planning workloads. The original extended logical-planning benchmark only calls logical optimization and does not exercise this physical schema helper. To run the added benchmark cases: ```sh cargo bench --profile release-nonlto -p datafusion-physical-plan --bench union_schema -- union_exec_try_new_non_nullable ``` ## Are there any user-facing changes? Faster construction of unions and interleaves with unequal schemas and non-nullable fields. Schema behavior and public APIs are preserved. AI assistance: OpenAI Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
