gene-bordegaray opened a new issue, #24724: URL: https://github.com/apache/datafusion/issues/24724
### Describe the bug - Filed from https://github.com/apache/datafusion/pull/24670 while fixing https://github.com/apache/datafusion/issues/24721 for @gene-bordegaray A cast to a data type and a cast to an explicit Arrow field have different metadata semantics: - A type-only cast should inherit metadata from the source field. - An explicit target field should use its own metadata, including an empty map that intentionally clears source metadata. DataFusion currently stores both forms as a target `FieldRef`. Once the target field has empty metadata, later planning cannot tell whether it was synthesized from a `DataType` or explicitly supplied. Cast target intent can therefore be lost during schema inference, logical-to-physical lowering, expression rewrites, and serialization. The user-visible effects include: - Explicit extension metadata, such as `arrow.uuid`, can disappear from cast output. - An explicitly empty target can incorrectly retain source extension metadata. - Standard SQL casts can incorrectly clear source metadata or retain unnecessary same-type casts. ### To Reproduce These two casts have different intended behavior but can produce the same stored target field: ```rust // Type-only: inherit source metadata. Cast::new(expr.clone(), DataType::FixedSizeBinary(16)); // Explicit field: clear source metadata. Cast::new_from_field( expr, Arc::new(Field::new("", DataType::FixedSizeBinary(16), true)), ); ``` If the source field contains: ```text metadata = {"ARROW:extension:name": "arrow.uuid"} ``` the first cast should retain `arrow.uuid`, while the second should produce an empty metadata map. Inspecting only `target_field.metadata().is_empty()` cannot distinguish these cases. The metadata-bearing form is visible with a custom SQL type planner: ```sql SELECT CAST(raw AS UUID), arrow_metadata(CAST(raw AS UUID), 'ARROW:extension:name') FROM ( VALUES ( arrow_cast(X'00010203040506070809000102030506', 'FixedSizeBinary(16)') ) ) AS uuids(raw); ``` The expected metadata value is `arrow.uuid`, not `NULL`. ### Expected behavior Cast target intent should be represented explicitly and preserved end to end: - Type-only casts inherit source field metadata. - Explicit target fields replace source metadata, even when their metadata map is empty. - Built-in SQL and standard Substrait casts use type-only targets. - Fields returned by a custom SQL `TypePlanner` remain explicit targets. - Logical and physical plan rewrites and protobuf round trips preserve the distinction. ### Additional context This was exposed while preserving `ProjectionExec` metadata in https://github.com/apache/datafusion/pull/24670. Some projection rewrites previously rederived the cast schema and accidentally masked the incorrect cast metadata contract. Once projection metadata was preserved correctly, the cast issue became observable. Using metadata emptiness as a sentinel is insufficient because an explicitly empty target field is valid and semantically different from a type-only target. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
