XiangpengHao commented on code in PR #8887:
URL: https://github.com/apache/arrow-rs/pull/8887#discussion_r2550717845
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -208,6 +233,21 @@ fn shredded_get_path(
return Ok(ArrayRef::from(target));
};
+ // Try to return the typed value directly when we have a perfect shredding
match.
+ if !matches!(as_field.data_type(), DataType::Struct(_)) {
+ if let Some(typed_value) = target.typed_value_field() {
+ let types_match = typed_value.data_type() == as_field.data_type();
+ let value_not_present = target.value_field().is_none();
+ // this is a perfect shredding, where the value is entirely
shredded out, so we can just return the typed value
+ // note that we MUST check value_not_present, because some of the
`typed_value` might be null but data is present in the `value` column.
+ // an alternative is to count whether typed_value has any
non-nulls, or check every row in `value` is null,
+ // but this is too complicated and might be slow.
Review Comment:
Thank you @scovich this makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
