Fokko commented on code in PR #4292:
URL: https://github.com/apache/iceberg-python/pull/4292#discussion_r4554203307
##########
pyiceberg/table/__init__.py:
##########
@@ -2813,7 +2834,14 @@ def plan_manifest_entries(self, manifests:
Iterable[ManifestFile]) -> Iterator[l
# this filter depends on the partition spec used to write the manifest
file
partition_evaluators: dict[int, Callable[[DataFile], bool]] =
KeyDefaultDict(self._build_partition_evaluator)
- metrics_evaluator = self._build_metrics_evaluator()
+ evaluate_metrics = self._build_metrics_evaluator()
+ metrics_evaluator = evaluate_metrics
+ if not self.prune_delete_files:
+
+ def keep_delete_files(data_file: DataFile) -> bool:
+ return data_file.content != DataFileContent.DATA or
evaluate_metrics(data_file)
+
+ metrics_evaluator = keep_delete_files
Review Comment:
Since a manifest holds either data or delete files, could we pick the
evaluator per manifest instead of checking `content` on every entry? That also
avoids having both `evaluate_metrics` and `metrics_evaluator`:
```python
metrics_evaluator = self._build_metrics_evaluator()
...
(
self.io,
manifest,
partition_evaluators[manifest.partition_spec_id],
metrics_evaluator if self.prune_delete_files or manifest.content ==
ManifestContent.DATA else lambda _: True,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]