mbutrovich opened a new issue, #4299:
URL: https://github.com/apache/iceberg-python/issues/4299

   ### Feature Request / Improvement
   
   Scan planning reads the manifest list into `ManifestFile` objects and then 
filters them by partition summaries 
([`table/__init__.py#L2805-L2809`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/table/__init__.py#L2805-L2809)).
 #4007 shows that `pyiceberg-core` parses a manifest list faster than Cython, 
but building the `ManifestFile` objects makes the end-to-end path slower.
   
   Apply the approach from #TBD-scan to `read_manifest_list`: read the list 
from `pyiceberg-core` as Arrow, and filter it on the Arrow columns so that only 
matching manifests become Python objects. This needs apache/iceberg-rust#3649. 
`pyiceberg-core` is an optional extra, so the Cython reader stays as the 
fallback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to