gene-bordegaray opened a new pull request, #24725:
URL: https://github.com/apache/datafusion/pull/24725
## Which issue does this PR close?
- Closes #24724.
## Rationale for this change
A cast to a data type and a cast to an explicit Arrow field have different
metadata semantics:
- A type-only cast should inherit metadata from the source field.
- An explicit target field should use its own metadata, including an empty
map that intentionally clears source metadata.
DataFusion currently stores both forms as a target `FieldRef`. Once that
field has empty metadata, planning cannot determine whether it was synthesized
from a `DataType` or explicitly supplied. This can lose extension metadata,
incorrectly retain metadata that should be cleared, or retain unnecessary
same-type casts.
## What changes are included in this PR?
- Introduce `CastTarget::{DataType, Field}` to represent cast intent
explicitly.
- Preserve the distinction through logical schema inference, physical
planning, expression rewrites, and protobuf serialization.
- Keep built-in SQL casts and standard Substrait casts type-only.
- Preserve fields returned by a custom SQL `TypePlanner` as explicit
targets, including nested custom fields and explicitly empty metadata.
- Decode legacy protobuf casts that do not contain `target_field`.
This is the prerequisite cast fix for #24670. The projection optimization
fix remains in that PR and is stacked on this branch.
## Are these changes tested?
Yes.
- `cargo test -p datafusion-sql`
- `cargo test -p datafusion-substrait`
- `cargo test --test sqllogictests` — 504/504 files passed with this
cast-only branch
- `cargo clippy -p datafusion-sql -p datafusion-substrait --all-targets
--all-features -- -D warnings`
- Focused logical schema, physical cast, logical/physical protobuf
round-trip, legacy decode, SQL type-planner, and Substrait cast tests
- `cargo fmt --all`
- `git diff --check`
## Are there any user-facing changes?
Cast output now consistently inherits or replaces Arrow field metadata
according to the requested target form. This fixes incorrect extension metadata
and unnecessary same-type casts.
This changes the public `Cast::field` and `TryCast::field` fields from
`FieldRef` to `CastTarget`. Callers that construct or inspect these structs
directly must handle the new enum. The existing `new` and `new_from_field`
constructors preserve their intended distinction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
