alexanderbianchi opened a new pull request, #27538: URL: https://github.com/apache/datafusion/pull/27538
## Which issue does this PR close? Related to #27517 and #14632. This is an exploratory architectural alternative to #27099, motivated by its discussion of resolving struct names before constructing DataFusion expressions. It does not supersede that PR or close the full issue. ## Rationale for this change Substrait omits struct field names from expression types. A column can therefore become `List<Struct<a>>` while a literal beside it becomes `List<Struct<c0>>`, causing an otherwise valid `coalesce`, `CASE`, or set operation to fail during consumption. This draft explores resolving those names before plan construction, including when the naming information comes from a later sibling or must cross projections and relation boundaries. ## What changes are included in this PR? - Build a per-conversion graph of nesting shapes and field-name variables from the Substrait tree. References share their producer's shape, and operations contribute explicit compatibility constraints. - Resolve names across `CASE` results, casts, corresponding set-operation outputs, and `IN` options. Projection and emit mappings preserve output lineage through supported relations. - Recognize the registered built-in `coalesce` implementation and explicit array/element signature relationships. Equal shapes alone do not connect independent arguments, and a user-defined replacement named `coalesce` is left independent. - Carry the immutable annotations through `SubstraitConsumerContext` extensions. Construct literals and cast targets with the resolved names, while retaining DataFusion's existing type coercion. This is an AI-assisted prototype for architectural discussion. Remaining limitations/design questions: - Lambda/subquery scopes, additional relation/expression forms, and opaque UDF relationships are unfinished. Custom relation consumers do not yet have an inference hook. - Annotations use protobuf node addresses guarded by a conversion lifetime token. Stable IDs in a prepared intermediate representation remain a design question. - Conflicting names retain the generated-name fallback and existing planning diagnostics. - The extra traversal has not been benchmarked; this PR makes no performance claim. ## What is the testing strategy for this PR? Twelve public-API tests in `struct_type_resolution.rs` check both output schemas and explicit expected rows. They cover later-sibling and nested `CASE` results, nested struct/list fields, typed-NULL casts in unions, limits/sorts, volatile projections, nested distinct unions, `coalesce`, and array element compatibility. Guards cover independent arguments and an overridden `coalesce`. With the implementation removed against local main (`73e0b0339`), ten tests fail and the two independence guards pass. All twelve pass with this implementation. Passed locally: - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-substrait` (314 tests/doctests passed) - `RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` (13,452 tests passed, plus all 307 SQL logic files) ## Are there any user-facing changes? Covered Substrait plans that previously failed because anonymous struct names differed now load and execute. This is incomplete support and remains a draft. No public Rust signatures change. The behavior of `from_substrait_rel`, `from_cast`, and the default `SubstraitConsumer::consume_literal` changes to prepare or use resolved names; all new implementation types are internal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
