lszskye commented on PR #8782: URL: https://github.com/apache/paimon/pull/8782#issuecomment-5044020881
> Could we preserve the value type from the data schema when constructing the selected-key read field here? > > Currently, this branch adds `field` directly to `readDataFields`. However, `field` is the temporary selected-key `ROW` using the current table schema’s MAP value type. For files written before a MAP value-type evolution, `SchemaEvolutionUtil.createIndexCastMapping` therefore sees identical expected/read types and does not create the required cast. > > I reproduced this with the following sequence: > > 1. Create a shared-shredding `MAP<STRING, INT>` column. > 2. Write a file containing `key1 -> 10`. > 3. Alter `metrics.value` from `INT` to `BIGINT`. > 4. Read `metrics['key1']` through selected-key pushdown. > > The ORC reader returns `0` instead of `10`, while the Parquet reader fails with `field 0 is missing`. > > The selected-key read `ROW` should use the MAP value type from `dataField` so that the physical reader reads the old file schema correctly, and the existing schema-evolution cast mapping can then convert that `ROW` to the current table value type. It would also be good to add an ORC/Parquet regression test covering this schema-evolution case. Thank you for pointing this out. You are right—there was indeed a correctness issue in this scenario. I have fixed it and added ORC and Parquet regression tests covering MAP value-type evolution with selected-key pushdown. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
