mbutrovich commented on PR #3467:
URL: https://github.com/apache/iceberg-rust/pull/3467#issuecomment-6958200413

   > What do you think about picking up the fix from upstream 
arrow-rs/parquet-rs instead, and keeping #3393 open until then?
   
   I agreed with this earlier, and I'd like to reopen it. arrow 60.1.0 shipped 
apache/arrow-rs#11657, and main has locked `arrow-schema` 60.1.0 since #3257. 
The reproduction from #3393 still fails on main at `7cf825fcf`. I ran it with a 
`ListView` and a `LargeListView` column, each read three ways. The results were 
the same for both types:
   
   | File | Project `id` | Project `lv`, `id` | Project `lv` |
   | --- | --- | --- | --- |
   | Field IDs at every level | correct | error | error |
   | No field IDs, name mapping | correct | `lv` all NULL | error |
   | No field IDs, fallback IDs | correct | error | error |
   
   The upstream fix only repaired the leaf shift, so projecting `id` alone now 
works. With field IDs, the mask builder now keeps the list view and fails 
converting it in 
[`arrow_schema_to_schema`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/arrow/schema.rs#L379)
 (`Cannot visit Arrow data type: ListView(...)`). The fallback and name-mapping 
errors come from 
[`RecordBatchTransformer`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/arrow/record_batch_transformer.rs#L372)
 (`cannot promote ListView(...) to List(...)`). The name-mapped `lv, id` case 
returns wrong results without an error.
   
   Since #3629 we rewrite `FixedSizeList` hints to `List` in 
`coerce_fixed_size_lists`, which follows this PR's design. That answers the 
`FixedSizeList` maintenance concern: both types can go through one rewrite. 
When I added `ListView` and `LargeListView` arms to that function, all 18 cases 
read correctly and the `arrow::` tests passed.
   
   The rewrite also keeps the reader correct for a downstream crate that 
resolves `arrow-schema` 60.0.0, which our `arrow-schema = "60.0"` requirement 
allows.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to