mbutrovich opened a new issue, #3649: URL: https://github.com/apache/iceberg-rust/issues/3649
### Is your feature request related to a problem or challenge? PyIceberg's scan planning filters every manifest by its partition summaries and every manifest entry with partition and metrics evaluators. Returning Arrow from `pyiceberg-core` (#TBD-binding) doesn't help it unless that filtering also runs on Arrow. In [`expr/visitors`](https://github.com/apache/iceberg-rust/tree/832a4eba7a732baebbbd7f76bb6fd6225f1db1e9/crates/iceberg/src/expr/visitors), `ManifestEvaluator` takes a parsed `ManifestFile`, and `ExpressionEvaluator` and `InclusiveMetricsEvaluator` take a parsed `DataFile`. Filtering with them would bring back the slower `ManifestList::parse_with_version` and `Manifest::parse_avro` decodes. This is Task 4 of #3262. ### Describe the solution you'd like Filter the output of `read_manifest_list` and `read_manifest_entries` on its Arrow columns, so that only matching rows become Python or Rust objects. Whether that runs vectorized in PyIceberg or in an Arrow-based evaluator in Rust is open, and this issue is the place to settle it. Once iceberg-rust's own scan planning filters over Arrow too, it can read manifests through one Avro path instead of both `Manifest::parse_avro` and `read_manifest_entries`, as @kevinjqliu suggested in the #3388 review. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
