alamb commented on code in PR #7661:
URL: https://github.com/apache/arrow-rs/pull/7661#discussion_r2156696877
##########
parquet-variant/src/variant.rs:
##########
@@ -665,33 +669,133 @@ impl<'m, 'v> VariantList<'m, 'v> {
}
}
-/// Variant value. May contain references to metadata and value
+/// Represents a Parquet Variant
+///
+/// The lifetimes `'m` and `'v` are for metadata and value, respectively.
+///
+/// # Background
+///
+/// The [specification] says:
+///
+/// The Variant Binary Encoding allows representation of semi-structured data
+/// (e.g. JSON) in a form that can be efficiently queried by path. The design
is
+/// intended to allow efficient access to nested data even in the presence of
+/// very wide or deep structures.
+///
+/// Another motivation for the representation is that (aside from metadata)
each
+/// nested Variant value is contiguous and self-contained. For example, in a
+/// Variant containing an Array of Variant values, the representation of an
+/// inner Variant value, when paired with the metadata of the full variant, is
+/// itself a valid Variant.
+///
+/// When stored in Parquet files, Variant fields can also be *shredded*.
Shredding
+/// refers to extracting some elements of the variant into separate columns for
+/// more efficient extraction/filter pushdown. The [Variant Shredding
Review Comment:
I checked the rendered version and it seems to be working. I am not sure
what you mean here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
