mbutrovich opened a new pull request, #3388: URL: https://github.com/apache/iceberg-rust/pull/3388
## Which issue does this PR close? - Part of #3262 (Task 3). ## What changes are included in this PR? Adds `iceberg::arrow::manifest_to_record_batch`, which converts a parsed `Manifest` into a `RecordBatch` with one row per entry, so `pyiceberg-core` can return Arrow instead of one Python object per entry. Manifest lists and the binding follow in separate PRs. - The schema is the spec's [`manifest_entry`](https://github.com/apache/iceberg/blob/5e7169168db3d34e29354c6f59ec4d6e420b8d2d/format/spec.md?plain=1#L685-L694) with the latest [`data_file`](https://github.com/apache/iceberg/blob/5e7169168db3d34e29354c6f59ec4d6e420b8d2d/format/spec.md?plain=1#L714-L743) fields and their field IDs, so every format version converts to the same schema apart from `data_file.partition`. - Building from a parsed `Manifest` gives the batch the same validation as `Manifest::parse_avro`. Bounds use the binary single-value serialization. - Partition values are typed by the partition spec, and a value that doesn't fit returns `DataInvalid`. On a 10,000-entry PyIceberg manifest, the conversion takes 5.5 ms on top of 36.7 ms for `Manifest::parse_avro`, in a release build on an Apple M5 Max. ## Are these changes tested? Yes. Unit tests in `arrow/manifest.rs` check the schema against the spec, every column for data and delete files, partition values of every primitive type and several transforms, and the error cases. The rustdoc example runs as a doctest. ## AI Disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
