alamb commented on code in PR #12311:
URL: https://github.com/apache/arrow-rs/pull/12311#discussion_r4643441651
##########
parquet-variant-compute/src/shred_variant.rs:
##########
@@ -401,32 +417,44 @@ impl<'a> VariantToShreddedObjectVariantRowBuilder<'a> {
return Ok(false);
};
- // Route the object's fields by name as either shredded or unshredded
+ // Route the object's fields by name as either shredded or unshredded.
An unshredded field
+ // keeps its field id and its encoded bytes, because the leftover
`value` column uses the
+ // same metadata dictionary.
let Self {
value_builder,
typed_value_builders,
+ sorted_field_names,
seen,
..
} = self;
seen.fill(false);
+ let object = obj.view();
Review Comment:
It is somewhat confusing that there is both `obj` (the view) and `object` in
this sciopt
it might be easier to read either to redefine `obj` or use `obj_view` or
something more distinguishing
##########
parquet-variant/src/builder/object.rs:
##########
@@ -151,6 +151,25 @@ impl<'a, S: BuilderSpecificState> ObjectBuilder<'a, S> {
Ok(())
}
+ /// Adds the field named by entry `field_id` of the metadata dictionary,
copying the encoded
+ /// `value` as is.
+ ///
+ /// The caller must ensure that `field_id`, and every field id inside
`value`, belong to the
+ /// metadata dictionary that this builder writes against. Copying a field
out of an object that
+ /// uses that same dictionary, as shredding does, satisfies this.
Review Comment:
```suggestion
/// The caller must ensure that `field_id`, and every field id inside
`value`, belong to the
/// builder's metadata dictionary. For example, during shredding, when
copying fields to the
/// output (residual) variant that are not shredded
```
##########
parquet-variant-compute/src/unshred_variant.rs:
##########
@@ -634,14 +642,18 @@ impl<'a> StructUnshredVariantBuilder<'a> {
));
};
- for entry in object.iter_try() {
- let (field_name, field_value) = entry?;
- if self.field_unshredders.contains_key(field_name) {
+ // The leftover fields keep their field ids and encoded bytes,
because the output uses
+ // the same metadata dictionary.
+ let object = object.view();
+ for i in 0..object.len() {
+ let field_name = object.field_name(i)?;
+ if self.sorted_field_names.binary_search(&field_name).is_ok() {
return Err(ArrowError::InvalidArgumentError(format!(
- "Field '{field_name}' appears in both typed_value and
value",
+ "Field '{}' appears in both typed_value and value",
+ String::from_utf8_lossy(field_name),
)));
}
- object_builder.insert_bytes(field_name, field_value);
+ object_builder.insert_encoded(object.field_id(i)?,
object.field(i)?.bytes()?)?;
Review Comment:
I think this is one key optimization -- just copy the original variant bytes
rather than re-walking/re-encoding a parsed one 👍
##########
parquet-variant/src/variant/view.rs:
##########
@@ -285,6 +291,40 @@ impl<'m, 'v> VariantView<'m, 'v> {
};
Variant::try_new_with_metadata_and_shallow_validation(metadata,
self.value)
}
+
+ /// The exact encoding of this value, without any bytes that follow it.
Review Comment:
the "without any bytes that follow it" is somewhat strange phrasing to me
It might be clearer if it just says "returns just the bytes that encode this
variant value"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]