JingsongLi commented on PR #8782:
URL: https://github.com/apache/paimon/pull/8782#issuecomment-5057452623
- When reading old files after a MAP field has been renamed,
`FormatReaderMapping.Builder.selectedKeysDataField` constructs the reader field
using `expectedField.newType(...)`, retaining the **new logical field name**.
- The footer and shredding metadata in old files are still stored using
the **old physical field names**.
- `MapSharedShreddingReadPlanFactory.shouldCreateReadPlan` looks up
metadata based on the reader field name; therefore, selected-key reads of old
files cannot create a shredding reconstruction plan, which may result in read
failures or incorrect results.
- Regular full MAP reads work because the original schema evolution path
retains the physical field names; only the new selected-key path loses them.
- It is recommended to construct the field based on `dataField` to
preserve the old name, then copy the selected-key description/type from
`expectedField`, for example,
`dataField.newType(...).newDescription(expectedField.description())`.
- A regression test for ORC/Parquet is needed to cover the scenario:
“write to the old field first → rename → selected-key read of the old file.”
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
