mbutrovich opened a new issue, #4298: URL: https://github.com/apache/iceberg-python/issues/4298
### Feature Request / Improvement `_open_manifest` ([`table/__init__.py#L2321-L2334`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/table/__init__.py#L2321-L2334)) filters each manifest entry with Python partition and metrics evaluators while decoding (#3658), but every entry still becomes a `DataFile` before it is filtered. #4007 shows that building those objects costs about as much as the whole Cython decode. To beat Cython, filtering has to happen on the Arrow columns from `pyiceberg-core`, so that only the matching entries become Python objects. This needs apache/iceberg-rust#3649, which settles whether the filtering runs vectorized in PyIceberg or in Rust. `pyiceberg-core` is an optional extra, so the Cython reader stays as the fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
