kosiew opened a new pull request, #16108:
URL: https://github.com/apache/datafusion/pull/16108
## Which issue does this PR close?
This is part of a series of PRs re-implementing #15295 to close #14657 by
adding schema‐evolution support for listing‐based tables with nested structs in
DataFusion.
## Rationale for this change
This refactor improves the flexibility and robustness of schema adaptation
in DataFusion by extracting casting logic into a reusable helper. It enhances
clarity, testability, and reusability of core logic for mapping file schemas to
table schemas—especially important for supporting schema evolution with nested
fields.
## What changes are included in this PR?
- Introduced `can_cast_field` to encapsulate field-level type casting logic
with clear error messaging.
- Added `create_field_mapping`, a helper for generating field mappings and
projections between file and table schemas.
- Refactored `DefaultSchemaAdapter::map_schema` to use
`create_field_mapping`, reducing duplication and improving readability.
- Added a `SchemaMapping::new` constructor for cleaner instantiation.
- Significantly expanded unit test coverage:
- Verified casting logic, including valid/invalid cast scenarios.
- Confirmed behavior of `create_field_mapping` under various mapping
strategies.
- Validated end-to-end schema mapping behavior via integration tests.
## Are these changes tested?
✅ Yes, comprehensive tests are included for:
- Field casting logic (`can_cast_field`)
- Field mapping creation (`create_field_mapping`)
- Full integration of schema adaptation via `map_schema` and
`SchemaMapping::map_batch`
These tests cover both happy-path and failure scenarios.
## Are there any user-facing changes?
No user-facing changes. This is an internal refactor that improves schema
handling logic and prepares the codebase for supporting complex schema
evolution use cases in a modular way.
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
