lyne7-sc opened a new issue, #3475:
URL: https://github.com/apache/iceberg-rust/issues/3475
### Apache Iceberg Rust version
main(acd610997de7)
### Describe the bug
Reading older Parquet files after a column rename can silently return values
under the wrong columns when the projection order differs from the file order
and the corresponding column types and nullability match.
For example, an orders table contains `paid_amount` and `refund_amount`,
both nullable DECIMAL(18, 2). After renaming `paid_amount` to `payment_amount`,
requesting `[refund_amount, payment_amount]` can return the payment amount as
the refund amount and vice versa.
The Parquet reader returns selected columns in file order.
`RecordBatchTransformer` can mistake the required reordering for a name-only
change because its schema comparison does not check field IDs.
### To Reproduce
1. Write a Parquet file with these fields in physical order:
- `paid_amount`: field ID 1, nullable DECIMAL(18, 2), value `100.00`.
- `refund_amount`: field ID 2, nullable DECIMAL(18, 2), value `20.00`.
2. Construct an Iceberg scan schema with `payment_amount` (ID 1) and
`refund_amount` (ID 2), preserving their types and nullability. Leave the
Parquet file unchanged.
3. Pass this schema to `FileScanTask` via `.with_schema(...)`, set
`.with_project_field_ids(vec![2, 1])`, and read through `ArrowReader`.
The output columns are `[refund_amount, payment_amount]`, but the values
remain `[100.00, 20.00]`.
### Expected behavior
The output values should follow the requested field IDs regardless of column
renames or physical column order.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
