linzch3 opened a new pull request, #3715:
URL: https://github.com/apache/iceberg-rust/pull/3715

   ## Which issue does this PR close?
   
   - Closes #3706.
   
   ## What changes are included in this PR?
   
   **Background.** If a struct's only file child is a Variant and that child is 
replaced by a new optional field, reading an old file together with a sibling 
column returns the whole struct as NULL, even for rows where it was present. 
This is the case #3438 left open.
   
   **Root cause.** `PresenceWalk` only accepts presence candidates whose path 
has field IDs. A Variant has its ID on the group, but its `metadata` and 
`value` leaves have none, so no candidate survives under the struct. The 
projection still isn't empty (the sibling is read), the struct is skipped, and 
`RecordBatchTransformer` fills it with NULL.
   
   **Fix.**
   - `PresenceWalk` now recognizes an unshredded Variant group by its file 
layout: an ID on the group, `metadata`/`value` bytes with no IDs (checked with 
`validate_unshredded_variant`). It goes by layout because the table schema may 
no longer have the Variant.
   - If the Variant isn't selected, its required `metadata` leaf becomes the 
presence candidate. It's the cheaper of the two leaves and is never null when 
the group is present.
   - If the Variant is selected, the group now counts as covering its parent, 
so no extra presence leaf gets read.
   - For the type check, the projected schema tags that group with the 
`arrow.parquet.variant` extension (the same as a projected Variant), so it 
folds into `Type::Variant` instead of tripping on the ID-less leaf.
   
   ## Are these changes tested?
   
   Yes.
   - `test_variant_dropped_child_preserves_parent_presence` is the repro from 
the issue, plus two more cases: a null parent row, and a present parent holding 
a null Variant. Each case checks that the struct's validity matches the file 
and that the new child is all NULL.
   - `test_selected_variant_covers_parent_presence` checks the `metadata` leaf 
is picked when the Variant isn't selected, and nothing is picked when it is.
   
   Both fail without the fix.
   
   Ran locally: `cargo test -p iceberg --lib` (2257 passed), `cargo clippy -p 
iceberg --all-targets -- -D warnings`, `cargo fmt --all -- --check`.
   
   **Limitations.**
   - Detection goes by layout. A plain struct with ID-less `metadata`/`value` 
binary children would be read as a Variant group here, but Iceberg writers 
always put IDs on struct children.
   - Shredded Variants (`typed_value`) are not matched and behave as before.
   
   ## AI Disclosure
   
   This change was prepared with an AI coding assistant; the tests above were 
run locally. Area of uncertainty: detecting the Variant by file layout rather 
than the Parquet `VARIANT` annotation, since older files may not carry it. I've 
also reviewed the changes myself.
   
   Opening as a draft because I'm at the open-PR limit for non-committers. 
Happy to rework the approach if you'd rather handle this differently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to