mbutrovich commented on code in PR #114:
URL: 
https://github.com/apache/datafusion-iceberg/pull/114#discussion_r4617642502


##########
crates/datafusion/src/read/scan.rs:
##########
@@ -93,6 +98,39 @@ impl IcebergDataSource {
     pub fn fetch(&self) -> Option<usize> {
         self.fetch
     }
+
+    /// Estimates the rows the scan returns before its predicate and fetch,
+    /// from the snapshot summary.
+    ///
+    /// `total-records` counts rows before deletes. A position delete removes
+    /// at most one row, so position deletes are subtracted. An equality delete
+    /// can remove any number of rows, so equality deletes are ignored. Summary
+    /// fields are optional. A missing or invalid `total-records` leaves the
+    /// estimate unknown, and a missing or invalid `total-position-deletes`
+    /// counts as zero, rather than failing the query.
+    fn estimated_num_rows(&self) -> Precision<usize> {
+        let metadata = self.table.metadata();
+        let snapshot = match self.snapshot_id {
+            Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),
+            None => match metadata.current_snapshot() {
+                Some(snapshot) => Some(snapshot),
+                // The scan of a table with no current snapshot reads nothing.
+                None => return Precision::Exact(0),
+            },
+        };
+        // The scan fails on a snapshot id that doesn't exist.
+        let Some(snapshot) = snapshot else {
+            return Precision::Absent;
+        };
+        let summary = &snapshot.summary().additional_properties;
+        let count = |key| summary.get(key)?.parse::<usize>().ok();
+        match count(TOTAL_RECORDS) {
+            Some(records) => Precision::Inexact(
+                
records.saturating_sub(count(TOTAL_POSITION_DELETES).unwrap_or(0)),

Review Comment:
   > Would it be safer to report `total-records` as-is, as @JanKaul's 
`datafusion_iceberg` does? Overestimating seems like the safer failure mode 
here.
   
   Agreed. The scan now reports `total-records` as an upper bound and ignores 
deletes. The spec lets 
[`total-position-deletes`](https://github.com/apache/iceberg/blob/9203eaff9eb97fef9d1f5a3ef4460e8453adaed9/format/spec.md?plain=1#L2495)
 count more rows than the deletes remove, in two ways:
   
   - No rule removes a v2 position delete file when the data files it 
references are removed. Only deletion vectors must go with their data file 
([spec](https://github.com/apache/iceberg/blob/9203eaff9eb97fef9d1f5a3ef4460e8453adaed9/format/spec.md?plain=1#L1713)).
 Your `RewriteDataFiles` case is the common one, since 
`remove-dangling-deletes` [defaults to 
`false`](https://github.com/apache/iceberg/blob/9203eaff9eb97fef9d1f5a3ef4460e8453adaed9/api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java#L109-L119).
   - A deletion vector must contain the deletes of existing position delete 
files, and readers then ignore those files 
([spec](https://github.com/apache/iceberg/blob/9203eaff9eb97fef9d1f5a3ef4460e8453adaed9/format/spec.md?plain=1#L1408),
 
[spec](https://github.com/apache/iceberg/blob/9203eaff9eb97fef9d1f5a3ef4460e8453adaed9/format/spec.md?plain=1#L1735)).
 Nothing requires removing a partition-scoped delete file that still applies to 
other data files, so those deletes are counted twice.
   
   
[`supports_collect_by_thresholds`](https://github.com/apache/datafusion/blob/500744d5c25499055e3b2ca46b022bc425989203/datafusion/physical-optimizer/src/join_selection.rs#L118-L139)
 compares inexact counts against the thresholds, so an underestimate there 
picks the large table as the build side. `ignores_deletes` covers a summary 
whose position deletes exceed its records.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to