mbutrovich opened a new issue, #3649:
URL: https://github.com/apache/iceberg-rust/issues/3649

   ### Is your feature request related to a problem or challenge?
   
   PyIceberg's scan planning filters every manifest by its partition summaries 
and every manifest entry with partition and metrics evaluators. Returning Arrow 
from `pyiceberg-core` (#TBD-binding) doesn't help it unless that filtering also 
runs on Arrow. In 
[`expr/visitors`](https://github.com/apache/iceberg-rust/tree/832a4eba7a732baebbbd7f76bb6fd6225f1db1e9/crates/iceberg/src/expr/visitors),
 `ManifestEvaluator` takes a parsed `ManifestFile`, and `ExpressionEvaluator` 
and `InclusiveMetricsEvaluator` take a parsed `DataFile`. Filtering with them 
would bring back the slower `ManifestList::parse_with_version` and 
`Manifest::parse_avro` decodes. This is Task 4 of #3262.
   
   ### Describe the solution you'd like
   
   Filter the output of `read_manifest_list` and `read_manifest_entries` on its 
Arrow columns, so that only matching rows become Python or Rust objects. 
Whether that runs vectorized in PyIceberg or in an Arrow-based evaluator in 
Rust is open, and this issue is the place to settle it.
   
   Once iceberg-rust's own scan planning filters over Arrow too, it can read 
manifests through one Avro path instead of both `Manifest::parse_avro` and 
`read_manifest_entries`, as @kevinjqliu suggested in the #3388 review.
   
   ### Willingness to contribute
   
   I can contribute to this feature independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to