kosiew opened a new pull request, #23914:
URL: https://github.com/apache/datafusion/pull/23914
## Which issue does this PR close?
Closes #20835
This is the last of a series of PR to close #20835
## Rationale for this change
Map types were not included in the existing recursive nested struct schema
adaptation logic, leaving Map key/value structs without consistent schema
evolution behavior. This change extends the central adaptation and validation
path to support Map types, defines the supported evolution semantics, and adds
regression coverage to ensure planner-time validation and runtime casting
remain consistent.
## What changes are included in this PR?
* Extend recursive nested struct casting to support `DataType::Map`.
* Add planner/runtime compatibility validation for Map schema evolution.
* Match Map entry children positionally (key, then value), while continuing
to match nested Struct fields by name.
* Implement recursive casting of Map keys and values, including:
* nullable field addition for supported struct evolution,
* omission of extra source fields,
* recursive nested struct adaptation.
* Validate Map-specific invariants, including:
* non-nullable entries and key fields,
* sorted Map compatibility rules,
* restrictions on Map key evolution to preserve key identity.
* Compact Map entries before casting when null parents or slicing leave
unreachable entries.
* Document Map evolution semantics in the nested compatibility helper.
* Add extensive unit tests covering supported and rejected Map evolution
scenarios.
* Add end-to-end Parquet tests covering successful Map value struct
evolution and rejection of incompatible schema evolution.
## Are these changes tested?
Yes.
This PR adds regression tests covering:
* Map entry names matched positionally while nested structs evolve by name.
* Map key and value struct schema evolution.
* Planner/runtime validation parity for supported and rejected evolution
paths.
* Sorted vs. unsorted Map behavior.
* Invalid key evolution scenarios.
* Null-parent and sliced Map handling during casting.
* All-null Map columns.
* Map entry invariant validation.
* End-to-end Parquet schema evolution for Map value structs, including both
successful adaptation and incompatible schema rejection.
These tests are added in:
* `datafusion/common/src/nested_struct.rs`
* `datafusion/core/tests/parquet/expr_adapter.rs`
## Are there any user-facing changes?
Yes.
Map types whose keys or values recursively contain Structs now participate
in nested schema evolution during schema adaptation. Supported evolution
follows the documented semantics implemented in this PR:
* Map entry children are matched by position (key, then value).
* Nested Struct fields are matched by name.
* Nullable target fields may be added and are populated with `NULL`.
* Extra source struct fields are omitted.
* Incompatible type changes, nullable-to-non-nullable changes, invalid Map
key evolution, and incompatible sorted Map changes are rejected consistently
during planning and execution.
No public API changes are introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
