Xuanwo opened a new pull request, #27502:
URL: https://github.com/apache/datafusion/pull/27502
## Which issue does this PR close?
Partially addresses #21982: this draft covers `make_array` and
`named_struct`, together with the argument-coercion behavior needed to keep
their schemas consistent. The issue also covers other constructors and
aggregates, so this PR does not close it.
Earlier attempts #21984 and #22176 are closed without merging. This draft
has a smaller constructor scope and also addresses metadata consistency across
implicit argument conversion.
## Rationale for this change
Wrapping a column in an array or named struct currently drops its
child-field metadata, including Arrow extension-type annotations. A consumer
can therefore receive a different logical type after a SQL projection even
though the underlying values were only wrapped in a container.
Preserving metadata also requires array arguments to share an exact item
type. Structs with identical child storage types can still differ in child
metadata or nullability; passing those different types to Arrow's array
concatenation can panic. The common type must be reflected in both the planned
schema and the actual input arrays.
For example, these constructors should preserve the declared child fields
and execute consistently when `name` and `l_name` are Utf8 columns carrying
different ordinary field metadata:
```sql
SELECT make_array(
named_struct('v', name),
named_struct('v', l_name)
)
FROM table_with_metadata;
```
## What changes are included in this PR?
- Preserve value fields in `named_struct`; derive `make_array`'s item field
from the coerced argument fields and use that field at execution time.
- Apply type-only cast metadata semantics in shared function argument
coercion and successful implicit literal materialization: preserve ordinary
metadata, remove both extension keys when the argument type changes, and retain
the complete field when its type is unchanged. Logical and physical type-only
casts use the same helper.
- Coerce struct array arguments to one exact item type. Child nullability is
combined across inputs; child metadata currently comes from the first coerced
struct, following existing list-coercion precedence.
The only new public item is `FieldMetadata::without_extension_type`, marked
`#[doc(hidden)]` and documented as internal to the workspace. Existing
supported public signatures are unchanged.
### Draft design questions
The first-input policy for struct child metadata needs further investigation
before this is ready for review. In particular, nested extension-name conflicts
are not explicitly reconciled, and the underlying struct coercion helper can
retain a child's extension annotation when that child's storage type changes.
The ordinary-metadata regression tests below do not establish correctness for
those nested extension cases.
Implicit literal conversion now drops extension annotations when its storage
type changes, matching column conversion. This is an observable semantic change
and is called out explicitly for review.
## What is the testing strategy for this PR?
Public SQL/DataFrame tests in `datafusion/core/tests/sql/select.rs` check
both planned and executed types, values and offsets, NULL inputs, parameter
metadata, type promotion, extension-name conflicts, and struct arguments with
different metadata/nullability in both argument orders. A focused coercion test
checks that only converted arguments lose extension annotations.
Local validation on commit `9beb06c145792b4ec772b0365a7f42d0b2603490`, based
on `120049489b5b74a3c8d942e5a97a1b7dbeec0ac2`:
- Formatting and `cargo clippy --all-targets --all-features -- -D warnings`
passed, using the CI profile with LTO disabled.
- Required extended workspace coverage completed across all 62 test
executables, including 307 SQLLogicTest files. The initial run hit `EMFILE` in
`test_sort_10k_mem` with a soft file-descriptor limit of 1024. Raising that
limit to 65535 and running the failed case and unreached targets completed the
coverage; successful cases from the same source were retained.
- The original metadata-difference SQL probe passes with the rebuilt
SQLLogicTest executable.
### Performance observations
Local `release-nonlto` measurements compare the commit above with its
unmodified base, using serial ABBA order, 12 pinned CPUs, and 30 samples per
case per round. The SQL workloads use 65,536 rows in 16 batches, with both
uniform non-null inputs and irregular values/nulls/string lengths.
Planning `make_array(a, a)` with a metadata-bearing Int32 column increased
from 21.161 to 22.010 microseconds (+4.01%). Planning `make_array(a, b)` with
Int32-to-Int64 promotion increased from 21.995 to 23.121 microseconds (+5.12%).
Full SQL-to-collect measurements across the constructor workloads changed by
-2.07% to +2.05%. These observations show a planning cost; they are not a claim
of an overall performance improvement.
## Are there any user-facing changes?
`named_struct` and `make_array` retain child-field metadata through planning
and execution. Struct array inputs with different ordinary metadata/nullability
are normalized to one type. Implicit literal storage conversion removes
extension annotations in the same way as column conversion, while preserving
ordinary metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]