rdblue commented on code in PR #461:
URL: https://github.com/apache/parquet-format/pull/461#discussion_r1809458948
##########
VariantShredding.md:
##########
@@ -33,140 +33,247 @@ This document focuses on the shredding semantics, Parquet
representation, implic
For now, it does not discuss which fields to shred, user-facing API changes,
or any engine-specific considerations like how to use shredded columns.
The approach builds upon the [Variant Binary Encoding](VariantEncoding.md),
and leverages the existing Parquet specification.
-At a high level, we replace the `value` field of the Variant Parquet group
with one or more fields called `object`, `array`, `typed_value`, and
`variant_value`.
-These represent a fixed schema suitable for constructing the full Variant
value for each row.
-
Shredding allows a query engine to reap the full benefits of Parquet's
columnar representation, such as more compact data encoding, min/max statistics
for data skipping, and I/O and CPU savings from pruning unnecessary fields not
accessed by a query (including the non-shredded Variant binary data).
Without shredding, any query that accesses a Variant column must fetch all
bytes of the full binary buffer.
-With shredding, we can get nearly equivalent performance as in a relational
(scalar) data model.
+With shredding, readers can get nearly equivalent performance as in a
relational (scalar) data model.
+
+For example, `SELECT variant_get(variant_event, '$.event_ts', 'timestamp')
FROM tbl` only needs to access `event_ts`, and the file scan could avoid
fetching the rest of the Variant value if this field was shredded into a
separate column in the Parquet schema.
+Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event,
'$.event_type', 'string') = 'signup'`, the scan could first decode the shredded
`event_type` column, and only fetch/decode the full Variant event value for
rows that pass the filter.
-For example, `select variant_get(variant_col, ‘$.field1.inner_field2’,
‘string’) from tbl` only needs to access `inner_field2`, and the file scan
could avoid fetching the rest of the Variant value if this field was shredded
into a separate column in the Parquet schema.
-Similarly, for the query `select * from tbl where variant_get(variant_col,
‘$.id’, ‘integer’) = 123`, the scan could first decode the shredded `id`
column, and only fetch/decode the full Variant value for rows that pass the
filter.
+## Variant Metadata
-# Parquet Example
+Variant metadata is stored in the top-level Variant group in a binary
`metadata` column regardless of whether the Variant value is shredded.
+All `variant_value` columns within the Variant must use the same `metadata`.
-Consider the following Parquet schema together with how Variant values might
be mapped to it.
-Notice that we represent each shredded field in `object` as a group of two
fields, `typed_value` and `variant_value`.
-We extract all homogenous data items of a certain path into `typed_value`, and
set aside incompatible data items in `variant_value`.
-Intuitively, incompatibilities within the same path may occur because we store
the shredding schema per Parquet file, and each file can contain several row
groups.
-Selecting a type for each field that is acceptable for all rows would be
impractical because it would require buffering the contents of an entire file
before writing.
+All fields for a variant, whether shredded or not, must be present in the
metadata.
Review Comment:
@sfc-gh-aixu, this is saying that when writing, the metadata for a shredded
value and the metadata for a non-shredded value should be identical. Writers
should not alter the metadata by removing shredded field names so that readers
do not need to rewrite the metadata (and values) to add it back.
For example, consider an event that looks like this:
```
{
"id": 102,
"event_type": "signup",
"event_timestamp": "2024-10-21T20:06:34.198724",
"payload": {
"a": 1,
"b": 2
}
}
```
And a shredding schema:
```
optional group event (VARIANT) {
required binary metadata;
optional binary value;
optional group typed_value {
required group event_type {
optional binary value;
optional binary typed_value (STRING);
}
required group event_timestamp {
optional binary value;
optional int64 typed_value (TIMESTAMP(true, MICROS));
}
}
}
```
The top-level `event_type` and `event_timestamp` fields are shredded. But
this is saying that the Variant `metadata` _must_ include those field names.
That ensure that the existing binary metadata can be returned to the engine
without adding `event_type` and `event_timestamp` fields when merging those
fields into the top-level Variant `value` when the entire Variant is projected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
