Fokko commented on code in PR #4278:
URL: https://github.com/apache/iceberg-python/pull/4278#discussion_r4553997285


##########
pyiceberg/table/__init__.py:
##########
@@ -2631,12 +2653,31 @@ def count(self) -> int:
         res = 0
         # every task is a FileScanTask
         tasks = self.plan_files()
+        bound_filter = bind(self.table_metadata.schema(), self.row_filter, 
case_sensitive=self.case_sensitive)
+        field_ids_outside_null_checks = visit(bound_filter, 
_FieldIDsOutsideNullChecks())
+        specs = self.table_metadata.specs() if field_ids_outside_null_checks 
else {}
+        referenced_positions = {
+            spec_id: [pos for pos, field in enumerate(spec.fields) if 
field.source_id in field_ids_outside_null_checks]
+            for spec_id, spec in specs.items()
+        }
 
         for task in tasks:
             # task.residual is a Boolean Expression if the filter condition is 
fully satisfied by the
             # partition value and task.delete_files represents that positional 
delete haven't been merged yet
             # hence those files have to read as a pyarrow table applying the 
filter and deletes
-            if task.residual == AlwaysTrue() and len(task.delete_files) == 0:
+            if (
+                task.residual == AlwaysTrue()
+                and len(task.delete_files) == 0
+                # A NULL partition value can leave an AlwaysTrue residual 
although the filter rejects its rows
+                and (
+                    not field_ids_outside_null_checks
+                    or (

Review Comment:
   I think this logic is problematic and hard to reason about. Why are we 
comparing the numer of partition fields? Feels awkward to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to