lszskye opened a new pull request, #8782:
URL: https://github.com/apache/paimon/pull/8782
## Purpose
Support reading selected keys directly from shared-shredding MAP columns.
Previously, queries such as:
SELECT attrs['key1'], attrs['key2'] FROM T
still read and reconstructed the complete MAP value. This PR connects
`__PAIMON_MAP_SELECTED_KEYS` with the shared-shredding reader so that only
the required physical columns and overflow data are read.
## Changes
- Recognize selected-key MAP read types in `FormatReaderMapping`.
- Build a pruned physical shared-shredding schema containing only:
- field mapping;
- physical columns that may contain the selected keys;
- overflow column when required.
- Materialize selected keys directly as a logical ROW without rebuilding
the complete MAP.
- Support missing keys, null MAP values, null values, dynamically mapped
physical columns, and overflow values.
- Validate that selected-key pushdown:
- targets a top-level MAP;
- uses the shared-shredding layout;
- preserves the complete MAP value type.
- Prevent changing the MAP storage layout of an existing field, because
files written with different layouts are not compatible.
- Preserve MAP storage-layout and max-columns options when renaming a column.
- Apply Spark selected-key rewriting during physical planning, after Spark
has created `DataSourceV2ScanRelation`.
- Add shared Spark E2E coverage for Spark 3.2, 3.3, 3.4, 3.5, 4.0, and 4.1.
## Tests
Added coverage for:
- selected physical-column schema pruning;
- reading selected keys from physical columns and overflow;
- missing and null keys;
- Parquet and ORC table reads;
- format-reader mapping and invalid selected-key types;
- MAP storage-layout changes and column renames;
- selected-key pushdown versus full MAP reads;
- normal MAP columns without shared shredding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
