sdf-jkl opened a new issue, #3706: URL: https://github.com/apache/iceberg-rust/issues/3706
### Apache Iceberg Rust version Upstream `main` at `4be78085219af9f1c2093bfcea66ff3d847c7613`; locked Arrow/Parquet 60.1.0. ### Describe the bug After replacing a struct's only Variant child with a new optional field, reading an older Parquet file alongside a sibling column returns NULL for the entire struct, even where that struct was present. This is a remaining case of #3378 after #3438. [`PresenceWalk::visit`](https://github.com/apache/iceberg-rust/blob/4be78085219af9f1c2093bfcea66ff3d847c7613/crates/iceberg/src/arrow/reader/projection.rs#L674-L755) discards candidates without field IDs. A valid Variant has an ID on its group, but its `metadata` and `value` leaves have no IDs, so neither can preserve the enclosing struct's presence. With another column selected, the projection is nonempty and the struct is omitted; batch conversion then fills it with NULL. Related: #3685. #3609 / #3624 address leaf numbering and absent-field projection; this case also needs presence selection to handle a Variant's ID-less storage leaves. ### To Reproduce 1. Write three rows with file schema `1: id int`, `2: s struct<3: old variant>`. Every `s` is present. 2. Read through the evolved schema `1: id int`, `2: s struct<4: new int?>`. 3. Project both `id` and `s`. Actual: `s = [NULL, NULL, NULL]`. Expected: `s = [{new: NULL}, {new: NULL}, {new: NULL}]`. The following test uses the existing Variant fixtures in `crates/iceberg/src/arrow/reader/projection.rs`; add it to that file's `tests` module: ```rust #[tokio::test] async fn review_variant_dropped_child_preserves_parent_presence() { use arrow_array::{Int32Array, StructArray}; let dir = TempDir::new().unwrap(); let children = Fields::from(vec![variant_arrow_field("old", 3)]); let source = Arc::new(ArrowSchema::new(vec![ Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(1)), Field::new("s", DataType::Struct(children.clone()), true).with_metadata(field_id_meta(2)), ])); let path = write_variant_parquet(source, vec![ Arc::new(Int32Array::from(vec![1,2,3])) as ArrayRef, Arc::new(StructArray::new(children, vec![variant_struct_array()], None)) as ArrayRef, ], dir.path().to_str().unwrap()); let target = Arc::new(Schema::builder().with_fields(vec![ NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), NestedField::optional(2, "s", Type::Struct(StructType::new(vec![ NestedField::optional(4, "new", Type::Primitive(PrimitiveType::Int)).into(), ]))).into(), ]).build().unwrap()); let batches = read_variant(target, &path, vec![1,2]).await; assert_eq!(batches[0].column(1).null_count(), 0, "present structs must survive replacement of their Variant child"); assert_eq!(batches[0].column(1).as_struct().column(0).null_count(), 3); } ``` Run `cargo test -p iceberg review_variant_dropped_child_preserves_parent_presence --lib`. The parent-null-count assertion fails with actual `3`, expected `0`. ### Expected behavior Preserve the file's parent struct validity while supplying NULL for the newly added optional child. This must also work when the available physical leaves under the parent are a Variant's ID-less storage fields. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
