Baunsgaard commented on code in PR #3658:
URL: https://github.com/apache/iceberg-rust/pull/3658#discussion_r4550425533


##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -329,6 +331,83 @@ pub(super) fn find_leaf_by_field_id(
     })
 }
 
+/// Parquet leaf index ranges of one map's key and value.
+struct MapLeafRanges {
+    key: Range<usize>,
+    value: Range<usize>,
+    value_is_struct: bool,
+}
+
+/// Collects map leaf ranges in post-order, recognising maps the way 
parquet-rs does.
+fn collect_map_leaf_ranges(ty: &ParquetType, next_leaf: &mut usize, maps: &mut 
Vec<MapLeafRanges>) {
+    if ty.is_primitive() {
+        *next_leaf += 1;
+        return;
+    }
+    let is_map = matches!(
+        ty.get_basic_info().converted_type(),
+        ConvertedType::MAP | ConvertedType::MAP_KEY_VALUE
+    );
+    if let (true, [entries]) = (is_map, ty.get_fields())
+        && !entries.is_primitive()
+        && let [key, value] = entries.get_fields()
+    {
+        let key_start = *next_leaf;
+        collect_map_leaf_ranges(key, next_leaf, maps);
+        let value_start = *next_leaf;
+        collect_map_leaf_ranges(value, next_leaf, maps);
+        maps.push(MapLeafRanges {
+            key: key_start..value_start,
+            value: value_start..*next_leaf,
+            value_is_struct: !value.is_primitive()
+                && value.get_basic_info().converted_type() == 
ConvertedType::NONE,
+        });
+    } else {
+        for field in ty.get_fields() {
+            collect_map_leaf_ranges(field, next_leaf, maps);
+        }
+    }
+}
+
+/// Parquet reads a map only when both its key and value are projected. When 
the key is selected
+/// but no value leaf matched the table (#3557), also read the value leaves: 
they match no table
+/// field and come back as a struct of NULLs (#3437). Only struct values are 
rescued, because a
+/// primitive value is matched by position and would be cast.
+fn add_map_value_leaves(
+    parquet_schema: &SchemaDescriptor,
+    indices: &mut Vec<usize>,
+    type_rejected_leaves: &HashSet<usize>,
+) -> Result<()> {
+    let mut maps = Vec::new();
+    collect_map_leaf_ranges(parquet_schema.root_schema(), &mut 0, &mut maps);
+    if maps.is_empty() {
+        return Ok(());
+    }
+    let mut selected: HashSet<usize> = indices.iter().copied().collect();
+    // Reversed post-order visits an outer map before the maps nested in its 
value.
+    for map in maps.iter().rev() {
+        let key_read = map.key.clone().any(|idx| selected.contains(&idx));
+        let value_read = map.value.clone().any(|idx| selected.contains(&idx));
+        if !key_read || value_read || !map.value_is_struct {
+            continue;
+        }
+        let value_leaves: Vec<usize> = map
+            .value
+            .clone()
+            .filter(|idx| !type_rejected_leaves.contains(idx))
+            .collect();

Review Comment:
   Confirmed the cast happens, but it doesn't come from the map rescue in this 
PR. On upstream/main, a list or map nested inside a struct map value whose 
element/key/value field id doesn't match the table **is still cast by 
position.** 
   
   The presence-leaf logic from #3438 reads that nested leaf, and the 
transformer then casts it. I've kept this PR scoped to #3557 and removed the 
nested-cast tests, since they would have locked in the current behaviour.
   
   Perhaps we should track a different issue for this ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to