goutamadwant opened a new pull request, #25010: URL: https://github.com/apache/datafusion/pull/25010
## Which issue does this PR close? - Addresses fix (1) in #24777. Constructor-based logical-plan decoding remains separate. ## Rationale for this change Column normalization repeatedly collects fallback schemas and traverses the input plan for USING columns. Wide expression lists and projections repeat that work for the same immutable plan. ## What changes are included in this PR? - Introduce a private, lazy normalization context reused across columns and expression lists. - Share the context in sort normalization and validated projection construction, including wildcard expansion. - Keep already-qualified columns and expressions that do not need normalization on the existing fast path. - Add benchmarks for qualified and unqualified expressions and projection construction at several schema widths. ## What is the testing strategy for this PR? - Add `normalize_batch_schema_precedence`, `normalize_batch_using_join`, and `normalize_batch_skips_unused_plan_context` to cover schema precedence, USING joins, ambiguity/error order, sort options, and lazy handling of qualified columns and literals. - In balanced local `release-nonlto` runs, constructing a 2,000-column unqualified projection falls from about 100 ms to 35 ms. This measures projection construction, not full protobuf decoding; small controls remain noisy. - Reproduce with `cargo bench -p datafusion-expr --bench normalize_columns --profile release-nonlto`. - Focused expression and SQL tests pass. The required extended workspace test command also passes, including all 511 SQL logic-test files. - Expression-crate Clippy passes with all targets and features enabled. The complete documented `dev/rust_lint.sh` also passes, including strict workspace documentation checks. - Full-workspace Clippy with all features enabled hits the existing PostgreSQL decimal-formatting lint in #24974; the affected source is unchanged here. ## Are there any user-facing changes? No public API or name-resolution behavior changes are intended. Normalization reuses plan context instead of collecting it for each column; existing schema lookup costs remain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
