JunRuiLee commented on code in PR #805:
URL: https://github.com/apache/paimon-rust/pull/805#discussion_r3976675124
##########
crates/paimon/src/table/data_file_reader.rs:
##########
@@ -943,6 +961,39 @@ fn prune_data_type(read_type: &DataType, data_type:
&DataType) -> crate::Result<
))))
}
}
+ // Java's `pruneDataType` descends ARRAY and MAP as well, and drops the
+ // container when nothing under it is selected, so a projection that
names
+ // only a child the file lacks reads as a NULL container rather than a
+ // container of NULLs.
+ DataType::Array(read_array) => {
+ let DataType::Array(data_array) = data_type else {
+ return Ok(Some(data_type.clone()));
+ };
+ let Some(element) =
+ prune_data_type(read_array.element_type(),
data_array.element_type())?
+ else {
+ return Ok(None);
+ };
+ Ok(Some(DataType::Array(
+ crate::spec::ArrayType::with_nullable(data_type.is_nullable(),
element),
Review Comment:
Confirmed. `vortex.rs` projects top-level names only
(`select(projected_names, root())`) and rebuilds the batch against the
requested Arrow type positionally, so a container pruned into read order
relabels its children.
Reverted the ARRAY/MAP descent. Containers keep their element type verbatim
and are reconciled after decoding by `evolve_column`, which pairs by field id;
two tests pin that with the reason so it doesn't get "aligned with Java" again.
The trade-off is the nullness difference from Java when a projection names only
a child the file lacks — noted in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
