anoopj commented on code in PR #3328:
URL: https://github.com/apache/iceberg-rust/pull/3328#discussion_r4234060872
##########
crates/iceberg/src/expr/visitors/page_index_evaluator.rs:
##########
@@ -394,14 +408,89 @@ impl<'a> PageIndexEvaluator<'a> {
Ok(page_filter)
}
+ ColumnIndexMetaData::FIXED_LEN_BYTE_ARRAY(idx) => {
+ // Parquet stores Iceberg decimals with precision > 18 as
+ // FIXED_LEN_BYTE_ARRAY. Other field types on such a column
+ // (fixed, uuid) aren't decoded here, so skip page pruning for
+ // them rather than risk pruning pages that match.
+ if !matches!(field_type, PrimitiveType::Decimal { .. }) {
+ tracing::debug!(
+ field_id,
+ %field_type,
+ "Skipping page-index pruning: FIXED_LEN_BYTE_ARRAY
column index on a non-decimal field"
+ );
+ return Ok(None);
+ }
+
+ let Some(type_length) = type_length else {
+ tracing::debug!(
+ field_id,
+ %field_type,
+ "Skipping page-index pruning: unknown
FIXED_LEN_BYTE_ARRAY decimal column width"
+ );
+ return Ok(None);
+ };
+
+ let mut page_filter = Vec::with_capacity(row_counts.len());
+ for ((i, (min, max)), &row_count) in idx
+ .min_values_iter()
+ .zip(idx.max_values_iter())
+ .enumerate()
+ .zip(row_counts.iter())
+ {
+ // A bound that won't decode (e.g. a min/max stat truncated
+ // by column_index_truncate_length, whose byte prefix no
+ // longer preserves the two's-complement decimal ordering)
+ // means this column's page index can't be trusted, so skip
+ // pruning for the whole column rather than abort the scan.
Review Comment:
Thanks. Hoisted the decode into a single Copy closure. Kept `Result<Datum>`
and `DataInvalid`: a short/undecodable bound isn't normal behavior for a
DECIMAL-annotated FLBA, so DataInvalid is the right signal. Happy to move both
to Option in a follow-up if you'd prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
