anuragmantri commented on code in PR #3424:
URL: https://github.com/apache/iceberg-rust/pull/3424#discussion_r4395387005
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -211,6 +248,27 @@ impl<'a> TableScanBuilder<'a> {
self
}
+ /// Retains the data file's column statistics for the named columns on
each planned
+ /// [`FileScanTask`], exposed as [`FileScanTask::column_stats`].
+ ///
+ /// Names resolve against the scan's schema with the scan's case
sensitivity and must name
+ /// primitive fields, which may be nested. By default, tasks carry no
statistics.
+ ///
+ /// [`build`](Self::build) fails if a name matches no field, matches more
than one field when
+ /// the scan is case-insensitive, or names a struct, list, map or metadata
column.
+ pub fn with_column_stats(
Review Comment:
Added. It is compile-only, because there is no table with data files to run
it against.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -760,6 +826,106 @@ impl From<FileScanTaskDeleteFile> for
Result<FileScanTaskDeleteFile> {
}
}
+/// Column statistics of a [`FileScanTask`]'s data file, keyed by field id and
limited to the
+/// columns requested for the scan.
+///
+/// A requested column missing from a map has unknown statistics of that kind,
for example when
+/// the table's metrics mode records no bounds. Bounds are inclusive even when
truncated, and are
+/// typed with the schema the manifest was written with, so after type
promotion an `int` bound
Review Comment:
Agreed, and thanks for the trace through `ManifestMergeManager`. Planning
now converts bounds to the column's type in the scan schema. The two-manifest
case is a test.
A decimal is converted only to a wider precision at the same scale. Any
other bound that differs in type is dropped, so it reads as unknown.
`date` to `timestamp` never gets this far. `Datum::try_from_bytes` fails on
the 4-byte bound, so the manifest read errors before planning. That seems like
a separate issue?
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -137,6 +138,12 @@ pub struct FileScanTask {
#[builder(default)]
sort_order: Option<SortOrderRef>,
+ /// The data file's column statistics for the columns requested with
+ ///
[`TableScanBuilder::with_column_stats`](crate::scan::TableScanBuilder::with_column_stats).
+ /// `None` if no statistics were requested.
+ #[builder(default)]
+ column_stats: Option<FileScanTaskColumnStats>,
Review Comment:
Done as you suggested. The setter and getter signatures are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
