c-thiel commented on code in PR #2880:
URL: https://github.com/apache/iceberg-rust/pull/2880#discussion_r4491836150
##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -2482,4 +3245,542 @@ message schema {
.unwrap();
assert_eq!(result, vec![expected]);
}
+
+ // ---------------------------------------------------------------------
+ // End-to-end variant read tests (synthetic Parquet).
+ //
+ // Unless a test says otherwise, the files carry a field id on the variant
group, none on
+ // its leaves, and no Parquet VARIANT annotation. `value` is required, as
in iceberg-java's
+ // unshredded layout.
+ // ---------------------------------------------------------------------
+
+ // Known variant bytes the reader must round-trip unchanged. `value` is
distinct per row,
+ // so a dropped or reordered row is caught, not just the struct shape.
+ const VARIANT_METADATA: [&[u8]; 3] = [b"\x01", b"\x01", b"\x01"];
+ const VARIANT_VALUE: [&[u8]; 3] = [b"\x0a", b"\x0b", b"\x0c"];
+
+ /// The 3-row variant `StructArray` for
[`VARIANT_METADATA`]/[`VARIANT_VALUE`].
+ fn variant_struct_array() -> ArrayRef {
+ use arrow_array::{BinaryArray, StructArray};
+ let metadata = Arc::new(BinaryArray::from(VARIANT_METADATA.to_vec()))
as ArrayRef;
+ let value = Arc::new(BinaryArray::from(VARIANT_VALUE.to_vec())) as
ArrayRef;
+ Arc::new(StructArray::new(
+ variant_arrow_fields(),
+ vec![metadata, value],
+ None,
Review Comment:
Thanks, good point! Added test_read_variant_with_null_rows in 524f5fddc. It
writes the iceberg-java layout (required metadata and value in an optional
group) with the middle row null, so the read goes through apply_struct. The
null row stays null and the other rows keep their bytes. I also checked that
the test fails when apply_struct drops source.nulls().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
