andygrove commented on code in PR #114:
URL:
https://github.com/apache/datafusion-iceberg/pull/114#discussion_r4615836268
##########
crates/datafusion/src/read/scan.rs:
##########
@@ -93,6 +98,39 @@ impl IcebergDataSource {
pub fn fetch(&self) -> Option<usize> {
self.fetch
}
+
+ /// Estimates the rows the scan returns before its predicate and fetch,
+ /// from the snapshot summary.
+ ///
+ /// `total-records` counts rows before deletes. A position delete removes
+ /// at most one row, so position deletes are subtracted. An equality delete
+ /// can remove any number of rows, so equality deletes are ignored. Summary
+ /// fields are optional. A missing or invalid `total-records` leaves the
+ /// estimate unknown, and a missing or invalid `total-position-deletes`
+ /// counts as zero, rather than failing the query.
+ fn estimated_num_rows(&self) -> Precision<usize> {
+ let metadata = self.table.metadata();
+ let snapshot = match self.snapshot_id {
+ Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),
+ None => match metadata.current_snapshot() {
+ Some(snapshot) => Some(snapshot),
+ // The scan of a table with no current snapshot reads nothing.
+ None => return Precision::Exact(0),
+ },
+ };
+ // The scan fails on a snapshot id that doesn't exist.
+ let Some(snapshot) = snapshot else {
+ return Precision::Absent;
+ };
+ let summary = &snapshot.summary().additional_properties;
+ let count = |key| summary.get(key)?.parse::<usize>().ok();
+ match count(TOTAL_RECORDS) {
+ Some(records) => Precision::Inexact(
+
records.saturating_sub(count(TOTAL_POSITION_DELETES).unwrap_or(0)),
Review Comment:
Thanks for this. It's a nice, contained way to give join selection something
to work with, and the unit tests cover the summary edge cases well.
I'm a bit worried about subtracting `total-position-deletes`. After
`rewrite_data_files` without `remove-dangling-deletes` (the default in Spark),
`total-records` reflects the compacted files while `total-position-deletes` can
still count deletes against data files that are gone. The estimate can then
land far below the real row count, down to `Inexact(0)` with the
`saturating_sub`. Because `join_selection` uses `num_rows` against
`hash_join_single_partition_threshold_rows`, a large table could get picked as
the `CollectLeft` build side, which is worse than reporting unknown stats.
Would it be safer to report `total-records` as-is, as @JanKaul's
`datafusion_iceberg` does? Overestimating seems like the safer failure mode
here. If you'd like to keep the subtraction, maybe fall back to `total-records`
(or `Absent`) when the deletes make up a large share of the records? A unit
test for the "deletes exceed records" case could then assert the fallback
instead of `Inexact(0)`.
##########
crates/datafusion/src/read/scan.rs:
##########
@@ -93,6 +98,39 @@ impl IcebergDataSource {
pub fn fetch(&self) -> Option<usize> {
self.fetch
}
+
+ /// Estimates the rows the scan returns before its predicate and fetch,
+ /// from the snapshot summary.
+ ///
+ /// `total-records` counts rows before deletes. A position delete removes
+ /// at most one row, so position deletes are subtracted. An equality delete
+ /// can remove any number of rows, so equality deletes are ignored. Summary
+ /// fields are optional. A missing or invalid `total-records` leaves the
+ /// estimate unknown, and a missing or invalid `total-position-deletes`
+ /// counts as zero, rather than failing the query.
+ fn estimated_num_rows(&self) -> Precision<usize> {
+ let metadata = self.table.metadata();
+ let snapshot = match self.snapshot_id {
+ Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),
Review Comment:
+1 to resolving the snapshot once at construction. Could we file an issue
for it so it doesn't get lost, and link it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]