JingsongLi commented on PR #8782: URL: https://github.com/apache/paimon/pull/8782#issuecomment-5041529782
Could we preserve the value type from the data schema when constructing the selected-key read field here? Currently, this branch adds `field` directly to `readDataFields`. However, `field` is the temporary selected-key `ROW` using the current table schema’s MAP value type. For files written before a MAP value-type evolution, `SchemaEvolutionUtil.createIndexCastMapping` therefore sees identical expected/read types and does not create the required cast. I reproduced this with the following sequence: 1. Create a shared-shredding `MAP<STRING, INT>` column. 2. Write a file containing `key1 -> 10`. 3. Alter `metrics.value` from `INT` to `BIGINT`. 4. Read `metrics['key1']` through selected-key pushdown. The ORC reader returns `0` instead of `10`, while the Parquet reader fails with `field 0 is missing`. The selected-key read `ROW` should use the MAP value type from `dataField` so that the physical reader reads the old file schema correctly, and the existing schema-evolution cast mapping can then convert that `ROW` to the current table value type. It would also be good to add an ORC/Parquet regression test covering this schema-evolution case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
