alamb commented on code in PR #12311:
URL: https://github.com/apache/arrow-rs/pull/12311#discussion_r4643441651


##########
parquet-variant-compute/src/shred_variant.rs:
##########
@@ -401,32 +417,44 @@ impl<'a> VariantToShreddedObjectVariantRowBuilder<'a> {
             return Ok(false);
         };
 
-        // Route the object's fields by name as either shredded or unshredded
+        // Route the object's fields by name as either shredded or unshredded. 
An unshredded field
+        // keeps its field id and its encoded bytes, because the leftover 
`value` column uses the
+        // same metadata dictionary.
         let Self {
             value_builder,
             typed_value_builders,
+            sorted_field_names,
             seen,
             ..
         } = self;
         seen.fill(false);
+        let object = obj.view();

Review Comment:
   It is somewhat confusing that there is both `obj` (the view) and `object` in 
this sciopt
   it might be easier to read either to redefine `obj` or use `obj_view` or 
something more distinguishing 



##########
parquet-variant/src/builder/object.rs:
##########
@@ -151,6 +151,25 @@ impl<'a, S: BuilderSpecificState> ObjectBuilder<'a, S> {
         Ok(())
     }
 
+    /// Adds the field named by entry `field_id` of the metadata dictionary, 
copying the encoded
+    /// `value` as is.
+    ///
+    /// The caller must ensure that `field_id`, and every field id inside 
`value`, belong to the
+    /// metadata dictionary that this builder writes against. Copying a field 
out of an object that
+    /// uses that same dictionary, as shredding does, satisfies this.

Review Comment:
   ```suggestion
       /// The caller must ensure that `field_id`, and every field id inside 
`value`, belong to the
       /// builder's metadata dictionary. For example, during shredding, when 
copying fields to the
       /// output (residual) variant that are not shredded
   ```



##########
parquet-variant-compute/src/unshred_variant.rs:
##########
@@ -634,14 +642,18 @@ impl<'a> StructUnshredVariantBuilder<'a> {
                 ));
             };
 
-            for entry in object.iter_try() {
-                let (field_name, field_value) = entry?;
-                if self.field_unshredders.contains_key(field_name) {
+            // The leftover fields keep their field ids and encoded bytes, 
because the output uses
+            // the same metadata dictionary.
+            let object = object.view();
+            for i in 0..object.len() {
+                let field_name = object.field_name(i)?;
+                if self.sorted_field_names.binary_search(&field_name).is_ok() {
                     return Err(ArrowError::InvalidArgumentError(format!(
-                        "Field '{field_name}' appears in both typed_value and 
value",
+                        "Field '{}' appears in both typed_value and value",
+                        String::from_utf8_lossy(field_name),
                     )));
                 }
-                object_builder.insert_bytes(field_name, field_value);
+                object_builder.insert_encoded(object.field_id(i)?, 
object.field(i)?.bytes()?)?;

Review Comment:
   I think this is one key optimization -- just copy the original variant bytes 
rather than re-walking/re-encoding a parsed one 👍 



##########
parquet-variant/src/variant/view.rs:
##########
@@ -285,6 +291,40 @@ impl<'m, 'v> VariantView<'m, 'v> {
         };
         Variant::try_new_with_metadata_and_shallow_validation(metadata, 
self.value)
     }
+
+    /// The exact encoding of this value, without any bytes that follow it.

Review Comment:
   the "without any bytes that follow it" is somewhat strange phrasing to me
   
   It might be clearer if it just says "returns just the bytes that encode this 
variant value"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to