alexanderbianchi opened a new pull request, #27538:
URL: https://github.com/apache/datafusion/pull/27538

   ## Which issue does this PR close?
   
   Related to #27517 and #14632. This is an exploratory architectural 
alternative to #27099, motivated by its discussion of resolving struct names 
before constructing DataFusion expressions. It does not supersede that PR or 
close the full issue.
   
   ## Rationale for this change
   
   Substrait omits struct field names from expression types. A column can 
therefore become `List<Struct<a>>` while a literal beside it becomes 
`List<Struct<c0>>`, causing an otherwise valid `coalesce`, `CASE`, or set 
operation to fail during consumption.
   
   This draft explores resolving those names before plan construction, 
including when the naming information comes from a later sibling or must cross 
projections and relation boundaries.
   
   ## What changes are included in this PR?
   
   - Build a per-conversion graph of nesting shapes and field-name variables 
from the Substrait tree. References share their producer's shape, and 
operations contribute explicit compatibility constraints.
   - Resolve names across `CASE` results, casts, corresponding set-operation 
outputs, and `IN` options. Projection and emit mappings preserve output lineage 
through supported relations.
   - Recognize the registered built-in `coalesce` implementation and explicit 
array/element signature relationships. Equal shapes alone do not connect 
independent arguments, and a user-defined replacement named `coalesce` is left 
independent.
   - Carry the immutable annotations through `SubstraitConsumerContext` 
extensions. Construct literals and cast targets with the resolved names, while 
retaining DataFusion's existing type coercion.
   
   This is an AI-assisted prototype for architectural discussion. Remaining 
limitations/design questions:
   
   - Lambda/subquery scopes, additional relation/expression forms, and opaque 
UDF relationships are unfinished. Custom relation consumers do not yet have an 
inference hook.
   - Annotations use protobuf node addresses guarded by a conversion lifetime 
token. Stable IDs in a prepared intermediate representation remain a design 
question.
   - Conflicting names retain the generated-name fallback and existing planning 
diagnostics.
   - The extra traversal has not been benchmarked; this PR makes no performance 
claim.
   
   ## What is the testing strategy for this PR?
   
   Twelve public-API tests in `struct_type_resolution.rs` check both output 
schemas and explicit expected rows. They cover later-sibling and nested `CASE` 
results, nested struct/list fields, typed-NULL casts in unions, limits/sorts, 
volatile projections, nested distinct unions, `coalesce`, and array element 
compatibility. Guards cover independent arguments and an overridden `coalesce`.
   
   With the implementation removed against local main (`73e0b0339`), ten tests 
fail and the two independence guards pass. All twelve pass with this 
implementation.
   
   Passed locally:
   
   - `cargo fmt --all -- --check`
   - `cargo clippy --all-targets --all-features -- -D warnings`
   - `cargo test -p datafusion-substrait` (314 tests/doctests passed)
   - `RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples 
--exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib 
--tests --bins --features 
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` 
(13,452 tests passed, plus all 307 SQL logic files)
   
   ## Are there any user-facing changes?
   
   Covered Substrait plans that previously failed because anonymous struct 
names differed now load and execute. This is incomplete support and remains a 
draft.
   
   No public Rust signatures change. The behavior of `from_substrait_rel`, 
`from_cast`, and the default `SubstraitConsumer::consume_literal` changes to 
prepare or use resolved names; all new implementation types are internal.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to