Amogh-2404 opened a new pull request, #26201:
URL: https://github.com/apache/datafusion/pull/26201
## Which issue does this PR close?
- Closes #26178.
## Rationale for this change
`CAST(coalesce(non_null_col, lit) AS T)` under `UNION ALL` + aggregate fails
planning when the column carries Arrow field metadata (`PARQUET:field_id`):
```
Internal error: Physical input schema should be the same as the one converted
from logical input schema. Differences:
- field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs
(logical) {}.
```
`coalesce` on a `NOT NULL` column simplifies to the column. Child
projections then advertise that metadata; the Union's stored schema still has
`{}` from the pre-simplify coalesce field. Physical planning derives from the
simplified exprs, so the aggregate schema check fails.
## What changes are included in this PR?
`LogicalPlan::recompute_schema` for `Union` no longer keeps the old schema
whenever field count is unchanged. It re-derives via `Union::try_new` so
metadata/nullability track the current children. If width is unchanged but
`try_new` fails (loose-typed children before coercion), the previous schema is
kept; width mismatches still return the `try_new` error.
## What is the testing strategy for this PR?
- Integration: `datafusion/core/tests/sql/union_field_metadata.rs` — issue
query (`m=3`, `c=6`), controls (no metadata / `CAST(b)` / direct `b AS x` / no
union), mixed-metadata legs.
- Unit: `recompute_schema_refreshes_union_field_metadata`, loose-union
preserve, width-mismatch error.
Verified fail-before / pass-after on
`coalesce_cast_union_aggregate_preserves_field_metadata` by temporarily
reverting the `plan.rs` change.
Commands run:
```bash
cargo test -p datafusion-expr --lib
logical_plan::plan::tests::recompute_schema
cargo test -p datafusion --test core_integration
coalesce_cast_union_aggregate
cargo test -p datafusion --test core_integration union_
cargo test -p datafusion-optimizer --lib
cargo test -p datafusion --lib physical_planner
cargo test -p datafusion-sqllogictest --test sqllogictests metadata.slt
cargo clippy -p datafusion-expr -p datafusion --tests -- -D warnings
cargo fmt --all -- --check
```
Not run: full `./dev/rust_lint.sh`, full extended CI suite, full
sqllogictest corpus.
## Are there any user-facing changes?
No public API change. Queries that previously hit this internal error now
plan. Union output may correctly carry field metadata that a stale schema had
dropped after expression rewrites. CAST metadata inheritance is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
