zenfenan opened a new pull request, #3658:
URL: https://github.com/apache/iceberg-python/pull/3658
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
Closes #3657
# Rationale for this change
`_open_manifest()` currently deserializes all live manifest entries via
`fetch_manifest_entry()` and then applies partitions & metrics filters as a
second pass. This materialises the full list before discarding non-matching
entries.
This PR adds `ManifestFile#prune_manifest_entry()` that fuses filtering into
the deserialization loop, avoiding the intermediate list allocation for
non-matching entries.
| Total Entries | Entries Matching Filter | Two-pass Time | Fused Time |
Improvement |
|-----------------------|------------------------|----------------|-------------|--------------|
| 116k | 78 (0.07%) | 9.13s | 7.18s | -21.3% |
| 438k | 1 (~0%) | 35.12s | 27.81s | -20.8% |
| 823k | 4,962 (0.6%) | 54.38s | 41.38s | -23.9% |
| 18k | 269 (1.5%) | 1.23s | 0.90s | -27.0% |
| 8k | 185 (2.2%) | 0.57s | 0.40s | -30.2% |
## Are these changes tested?
Yes, existing test suit passes. No behavioral change. Correctness was
validated across all benchmark runs (matched entry counts identical between
both paths).
## Are there any user-facing changes?
No.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
