mbutrovich commented on code in PR #3302:
URL: https://github.com/apache/iceberg-rust/pull/3302#discussion_r4606315356


##########
crates/iceberg/src/arrow/reader/predicate_visitor.rs:
##########
@@ -195,6 +197,251 @@ impl BoundPredicateVisitor for CollectFieldIdVisitor {
     }
 }
 
+/// Returns the residual of `predicate` for one data file: each leaf on a 
top-level field that is
+/// missing from the file becomes `AlwaysTrue` or `AlwaysFalse`, based on the 
value projection
+/// returns for that field. That value is the identity partition value, 
otherwise the field's
+/// `initial-default`, and every row of the file holds it. Leaves on missing 
fields with neither
+/// keep the null handling of the row filter and the page index evaluator. The 
resulting constants
+/// are folded out of `AND` and `OR`.
+///
+/// Leaves on nested fields are kept, because a nested field also reads as 
null in any row where
+/// an ancestor struct is null.
+pub(super) fn residual_for_missing_fields(
+    predicate: BoundPredicate,
+    predicate_field_ids: &HashSet<i32>,
+    field_id_map: &HashMap<i32, usize>,
+    schema: &Schema,
+    partition_spec: Option<&PartitionSpec>,
+    partition: Option<&Struct>,
+) -> Result<BoundPredicate> {
+    if predicate_field_ids
+        .iter()
+        .all(|id| field_id_map.contains_key(id))
+    {
+        return Ok(predicate);
+    }
+
+    let partition_constants = match (partition_spec, partition) {
+        (Some(spec), Some(data)) => constants_map(spec, data, schema)?,

Review Comment:
   > The manifest is decoded with its own schema, so after `b` is promoted from 
int to long, `task.partition()` holds `Literal::int(7)` and `constants_map` 
builds `Datum::new(Long, Int(7))`.
   
   Thanks @JeonDaehong. #3483 fixed this at planning time after your review. 
The scan now [widens the partition tuple to the current 
schema](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/scan/context.rs#L148)
 when it reads a manifest, so `FileScanTask::partition()` holds 
`Literal::long(7)` for a file written before the promotion. 
[`test_scan_widens_old_partition_so_pruning_and_deletes_match`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/scan/mod.rs#L1902-L1906)
 checks that. I merged `main` in d3a3f7c1bbed1797d3543846c9d1e766464c8cae, so 
`constants_map` gets the widened value on this branch, and the residual and 
projection both see `long(7)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to