mbutrovich opened a new issue, #4299: URL: https://github.com/apache/iceberg-python/issues/4299
### Feature Request / Improvement Scan planning reads the manifest list into `ManifestFile` objects and then filters them by partition summaries ([`table/__init__.py#L2805-L2809`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/table/__init__.py#L2805-L2809)). #4007 shows that `pyiceberg-core` parses a manifest list faster than Cython, but building the `ManifestFile` objects makes the end-to-end path slower. Apply the approach from #TBD-scan to `read_manifest_list`: read the list from `pyiceberg-core` as Arrow, and filter it on the Arrow columns so that only matching manifests become Python objects. This needs apache/iceberg-rust#3649. `pyiceberg-core` is an optional extra, so the Cython reader stays as the fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
