patrickswedish commented on PR #24394:
URL: https://github.com/apache/datafusion/pull/24394#issuecomment-5358803618
Hi @kosiew,
Thank you very much for catching this edge case and for the detailed review!
We have implemented recursive adaptation for Union arrays and addressed all
your points:
1. **Recursive Union Adaptation (\cast_union_column\)**:
- In \datafusion_common::nested_struct\, implemented \cast_union_column\
supporting both \Sparse\ and \Dense\ union modes.
- Recursively casts each child array to the target child field's data
type using \cast_column\.
- Preserves row-level \ ype_ids\ buffer and dense \offsets\ buffer
without copying unnecessary primitive data.
- For extra fields present only in the target union, populates them with
empty arrays (Dense) or typed null arrays (Sparse).
- Validates that union modes match (\source_mode == target_mode\) and
that all source type IDs exist in the target union, cleanly returning a
deterministic planning error if mismatched.
2. **Integration with Cast and Validation**:
- Updated \cast_column\ to route \(DataType::Union, DataType::Union)\
through \cast_union_column\.
- Updated \ alidate_data_type_compatibility\ and \
equires_nested_struct_cast\ to validate matching union modes and recursively
validate child field compatibility.
3. **Comprehensive Regression Tests**:
- Direct unit tests in
\datafusion_common::nested_struct::adapt_schema_tests\:
- \ est_adapt_batch_to_schema_stricter_sparse_union\
- \ est_adapt_batch_to_schema_stricter_dense_union\
- \ est_adapt_batch_to_schema_union_nested_struct\
- \ est_adapt_batch_to_schema_union_incompatible_mode_rejected\
- \ est_adapt_batch_to_schema_union_missing_type_id_rejected\
- \ est_validate_data_type_compatibility_union\
- Direct \AggregateExec\ input boundary integration tests in
\datafusion/core/tests/sql/aggregates/nested_nullability.rs\:
- \ est_aggregate_exec_direct_input_adaptation_dense_union\
- \ est_aggregate_exec_direct_input_adaptation_sparse_union\
The branch has been rebased onto the latest \main\, formatted with \cargo
fmt\, and validated with \cargo clippy --all-targets --all-features -- -D
warnings\ and the full test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
