mbutrovich commented on code in PR #3424:
URL: https://github.com/apache/iceberg-rust/pull/3424#discussion_r4517489599
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -760,6 +826,106 @@ impl From<FileScanTaskDeleteFile> for
Result<FileScanTaskDeleteFile> {
}
}
+/// Column statistics of a [`FileScanTask`]'s data file, keyed by field id and
limited to the
+/// columns requested for the scan.
+///
+/// A requested column missing from a map has unknown statistics of that kind,
for example when
+/// the table's metrics mode records no bounds. Bounds are inclusive even when
truncated, and are
+/// typed with the schema the manifest was written with, so after type
promotion an `int` bound
Review Comment:
> `date` to `timestamp` never gets this far. `Datum::try_from_bytes` fails
on the 4-byte bound, so the manifest read errors before planning. That seems
like a separate issue?
Agreed, and it's already filed. #3387 covers the manifest read failing on a
4-byte bound under `timestamp` or `timestamp_ns`. #3503 covers promoting `date`
bounds written before the promotion. Until #3503 lands, a `date` bound from an
older manifest is dropped here, which reads as unknown. Since
[`bound_as_column_type`](https://github.com/apache/iceberg-rust/blob/86e35d3d6365fa3dc1a17be26096f45b2b3e6985/crates/iceberg/src/scan/task.rs#L870-L874)
tries `Datum::promote_to` first, a fix for #3503 that adds the `date` arms
there will reach these statistics with no change to this code.
I checked the promotion on 86e35d3:
`test_plan_files_promotes_bounds_to_scan_schema_type` writes one manifest with
`y: int` and one with `y: long`, and both tasks come back with `Datum::long`
bounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
