comphead opened a new pull request, #3396: URL: https://github.com/apache/iceberg-rust/pull/3396
## Which issue does this PR close? - Closes #3394. ## What changes are included in this PR? `get_arrow_projection_mask_with_field_ids` carried each leaf's field ID from its first pass over the Arrow leaves to the second through a `HashMap<FieldRef, i32>`. Hashing an Arrow `Field` hashes its whole `DataType` and allocates a sorted copy of its metadata keys, and the map grew from empty. The second pass now reads the ID from the leaf's `PARQUET:field_id` metadata, as the first pass already does, and the map is gone. `build_field_id_map_from_arrow_schema` uses the same small helper. Behavior does not change. One call to `ArrowReader::get_arrow_projection_mask` for a file whose columns are all `int` with field IDs (release build, Apple M3 Max, 500 calls each, two runs): | Columns | Projected | Allocations before | Allocations after | Time before | Time after | | ---: | ---: | ---: | ---: | ---: | ---: | | 100 | 10 | 486 | 172 | 34 to 46 µs | 13 µs | | 1,000 | 10 | 3,976 | 178 | 318 µs | 95 µs | | 1,000 | 1,000 | 15,915 | 12,117 | 897 µs | 657 to 660 µs | The numbers come from the local counting-allocator test in #3394, which is not part of this PR. This conflicts with #3384 on one line, the declaration of the map, which #3384 sizes up front and this PR removes. ## Are these changes tested? Existing tests cover the projection mask, such as `test_arrow_projection_mask` and the reader tests in `projection.rs`. `cargo test -p iceberg --lib` passes, and clippy is clean. ## AI Disclosure Drafted with Claude Code. The numbers come from the test in #3394, run against `main` at `b2e90754b` and this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
