mbutrovich opened a new pull request, #3652:
URL: https://github.com/apache/iceberg-rust/pull/3652
## Which issue does this PR close?
- Closes #3647.
- Closes #3648.
- Part of #3262 (Task 3).
## What changes are included in this PR?
- `read_manifest_entries` (#3388) and `read_manifest_list` (#3631) haven't
been released, so their signatures change without a deprecation.
- The batch size is 1024 rows. An entry's size grows with the table's column
count through the column stats maps. These are the Arrow sizes of 8 MB
manifests written by PyIceberg with stats on every long column. The stats
compress well, which fits the most entries in 8 MB.
| Table columns | Entries in an 8 MB manifest | Arrow size per entry | One
batch (`main`) | Largest batch of 8192 | Largest batch of 1024 |
|--:|--:|--:|--:|--:|--:|
| 12 | 61,173 | 1.4 KB | 85.2 MB | 10.5 MB | 1.3 MB |
| 100 | 9,935 | 7.3 KB | 71.2 MB | 70.0 MB | 8.8 MB |
| 1000 | 810 | 86 KB | 68.2 MB | 68.2 MB (whole manifest) | 68.2 MB (whole
manifest) |
Wider tables have larger entries, so fewer fit in an 8 MB manifest. At
1000 columns, a manifest holds fewer than 1024 entries and reads as one batch
at either size.
## Are these changes tested?
- Unit tests and doctests in `crates/iceberg/src/arrow/manifest.rs`.
- Python tests in `bindings/python/tests/test_manifest.py`, which compare
the binding's output with PyIceberg's own readers.
## AI Disclosure
I wrote this PR with help from Claude but understand and support the core
approach, implementation, and test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]