mbutrovich opened a new issue, #4007: URL: https://github.com/apache/iceberg-python/issues/4007
### Feature Request / Improvement PyIceberg reads manifests and manifest lists with its Cython Avro decoder. `pyiceberg-core` (the iceberg-rust Python binding) already exposes `read_manifest_entries` and `read_manifest_list`, but using them today makes manifest reads 4x to 5x slower than Cython. This issue tracks the work in both repos to make the `pyiceberg-core` path faster than Cython, and then to switch PyIceberg to it. The iceberg-rust side is tracked in apache/iceberg-rust#3262, which repeats those tasks in more detail. The earlier attempt (apache/iceberg-rust#1280, apache/iceberg-python#2493) was closed. It converted every Rust manifest entry into a PyIceberg `DataFile` in Python, which is the `rust_convert` path measured below. The full benchmark writeup from @kevinjqliu, including an in-crate profile of the Rust parser and the script, is in https://github.com/kevinjqliu/iceberg-python/issues/45. #### Where the time goes These numbers come from manifests written by PyIceberg (V2, deflate, 12 columns with full column stats, identity partition). `pyiceberg` is at `main` (0d58407) with the Cython decoder. `pyiceberg-core` is built locally from the `v0.10.1` tag, once as released (`opt-level = "z"`) and once with `opt-level = 3`. Each number is the best of 7 runs in ms, single threaded, on Apple Silicon with Python 3.12. `v0.10.1` predates apache/iceberg-rust#3028, which its PR measured as making the Rust parse about 5% faster. | entries | Cython | `pyiceberg-core` as released | `pyiceberg-core` with `opt-level = 3` | |--:|--:|--:|--:| | 1,000 | 8.7 | 43.9 | 30.8 | | 10,000 | 94.1 | 452.3 | 320.4 | | 50,000 | 565.9 | 2308.7 | 1628.0 | The `pyiceberg-core` columns include building PyIceberg `ManifestEntry` and `DataFile` objects, so all three columns produce the same result. At 10,000 entries with `opt-level = 3`, the Rust parse costs 21.7 us per entry and converting the result into PyIceberg objects costs another 10.3 us per entry. The whole Cython path costs 10.3 us per entry in the same run. The conversion alone costs as much as Cython, so a faster Rust parser can't reach parity unless PyIceberg also stops building one Python object per entry from the binding's output. Manifest lists are closer. With `opt-level = 3` the Rust parse of a 10,000-entry manifest list takes 8.0 ms against 23.9 ms for Cython, and the conversion into `ManifestFile` objects is what makes the end-to-end path slower (39.1 ms). #### Task list These tasks are in PyIceberg. - [x] **Write `equality_ids` as `list<int>` per the spec.** PR: #3842. - [ ] **Flush manifest entries in size-bounded Avro blocks.** `ManifestWriter.add_entry` calls `write_block` once per entry ([`manifest.py#L1213-L1238`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/manifest.py#L1213-L1238)), so every entry gets its own deflate stream and sync marker. Java writes manifests through Avro's `DataFileWriter` and keeps its default sync interval, which the linked issue reports as 64 KB. In the linked benchmark, a 10,000-entry manifest shrank from 3.6 MB to 0.7 MB and read about 1.5x faster with both decoders after re-encoding with 64 KB blocks. The manifest-list writer already writes one block ([`manifest.py#L1389`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/manifest.py#L1389)). This task doesn't depend on anything else. - [ ] **Build `inspect.entries()` and `inspect.files()` from `pyiceberg-core` Arrow output.** Both methods build a `DataFile` per entry and then flatten them back into a `pa.Table` ([`inspect.py#L152`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/table/inspect.py#L152), [`inspect.py#L860`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/table/inspect.py#L860)). These are the first consumer of the Arrow output from the iceberg-rust tasks, because they want Arrow anyway. This task depends on that output being in a `pyiceberg-core` release. - [ ] **Plan scans through `pyiceberg-core`.** `_open_manifest` ([`table/__init__.py#L2321-L2334`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/table/__init__.py#L2321-L2334)) filters each entry with Python partition and metrics evaluators while decoding (#3658), but every entry still becomes a `DataFile` before it is filtered. To beat Cython, filtering has to happen on Arrow columns or inside Rust, so that only the matching entries become Python objects. Which of the two to use is open and is settled together with the iceberg-rust task for filtering. `pyiceberg-core` is an optional extra, so the Cython reader stays as the fallback. - [ ] **Read manifest lists through `pyiceberg-core`.** This uses the same approach as scan planning, applied to `read_manifest_list`. These tasks are in iceberg-rust and tracked in apache/iceberg-rust#3262. - [ ] **Faster manifest parsing.** Upgrade to `apache-avro` 0.22 and decode entries directly from the writer schema instead of through `apache_avro::Value` and schema resolution (apache/iceberg-rust#3063). - [ ] **Build the `pyiceberg-core` wheel with `opt-level = 3`.** In the numbers above, this makes the parse about 1.6x faster and the end-to-end path about 1.4x faster. It also nearly doubles the wheel size. - [ ] **Return manifest entries and manifest lists as Arrow from `pyiceberg-core`.** - [ ] **Release `pyiceberg-core` with the above.** PyIceberg pins `pyiceberg-core>=0.10.1,<0.11.0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
