lszskye opened a new pull request, #8941:
URL: https://github.com/apache/paimon/pull/8941
### Purpose
Previously, Paimon rejected changing the storage layout of an existing MAP
field between `default` and `shared-shredding`.
This restriction also prevented tables from gradually migrating their MAP
storage layout. After enabling shared shredding, a table can contain both old
default-layout MAP files and newly written shared-shredding files.
Selected-key
pushdown must therefore be able to read both physical representations
correctly.
This PR:
- Allows an existing MAP field to switch between `default` and
`shared-shredding` while keeping the existing schema validation.
- Supports switching the layout in both directions:
- `default -> shared-shredding`
- `shared-shredding -> default`
- switching back to `shared-shredding`
- Creates a selected-key read plan even when an old file does not contain
shared-shredding metadata.
- Restores the physical MAP read type for selected-key reads of
default-layout
files.
- Projects the requested keys from a normal `MapColumnVector`, preserving the
semantics of:
- existing keys
- missing keys
- null MAP values
- keys whose values are null
- Keeps selected-key pushdown enabled only when the current table layout is
`shared-shredding`.
- Supports mixed-layout reads after MAP value-type evolution and column
rename.
- Verifies that forced compaction rewrites old shared-shredding files into
the
current default layout.
### Tests
Added and updated tests covering:
- `MapSharedShreddingUtils.buildPhysicalReadType` with:
- no shared-shredding metadata
- full MAP reads
- selected-key reads from default-layout files
- full and selected-key reads from shared-shredding files
- field IDs, descriptions and nullability
- Selected-key materialization from normal MAP column vectors, including
missing keys, null values and null MAP rows.
- Read-plan creation for selected-key reads of default-layout files.
- Schema changes in both directions, including changes combined with a column
rename.
- ORC and Parquet reads across mixed default and shared-shredding files.
- MAP value-type evolution from `INT` to `BIGINT`.
- Full MAP and selected-key reads before and after layout changes.
- Forced compaction after switching from shared shredding to the default
layout.
- Spark selected-key pushdown and result correctness across all layout
generations.
The relevant common and core unit tests pass. The Spark E2E suite passes for
Spark 3.2, 3.3, 3.4, 3.5, 4.0 and 4.1 with both ORC and Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
