rdblue commented on code in PR #461:
URL: https://github.com/apache/parquet-format/pull/461#discussion_r1815658988
##########
VariantShredding.md:
##########
@@ -33,140 +33,247 @@ This document focuses on the shredding semantics, Parquet
representation, implic
For now, it does not discuss which fields to shred, user-facing API changes,
or any engine-specific considerations like how to use shredded columns.
The approach builds upon the [Variant Binary Encoding](VariantEncoding.md),
and leverages the existing Parquet specification.
-At a high level, we replace the `value` field of the Variant Parquet group
with one or more fields called `object`, `array`, `typed_value`, and
`variant_value`.
-These represent a fixed schema suitable for constructing the full Variant
value for each row.
-
Shredding allows a query engine to reap the full benefits of Parquet's
columnar representation, such as more compact data encoding, min/max statistics
for data skipping, and I/O and CPU savings from pruning unnecessary fields not
accessed by a query (including the non-shredded Variant binary data).
Without shredding, any query that accesses a Variant column must fetch all
bytes of the full binary buffer.
-With shredding, we can get nearly equivalent performance as in a relational
(scalar) data model.
+With shredding, readers can get nearly equivalent performance as in a
relational (scalar) data model.
+
+For example, `SELECT variant_get(variant_event, '$.event_ts', 'timestamp')
FROM tbl` only needs to access `event_ts`, and the file scan could avoid
fetching the rest of the Variant value if this field was shredded into a
separate column in the Parquet schema.
+Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event,
'$.event_type', 'string') = 'signup'`, the scan could first decode the shredded
`event_type` column, and only fetch/decode the full Variant event value for
rows that pass the filter.
-For example, `select variant_get(variant_col, ‘$.field1.inner_field2’,
‘string’) from tbl` only needs to access `inner_field2`, and the file scan
could avoid fetching the rest of the Variant value if this field was shredded
into a separate column in the Parquet schema.
-Similarly, for the query `select * from tbl where variant_get(variant_col,
‘$.id’, ‘integer’) = 123`, the scan could first decode the shredded `id`
column, and only fetch/decode the full Variant value for rows that pass the
filter.
+## Variant Metadata
-# Parquet Example
+Variant metadata is stored in the top-level Variant group in a binary
`metadata` column regardless of whether the Variant value is shredded.
+All `variant_value` columns within the Variant must use the same `metadata`.
-Consider the following Parquet schema together with how Variant values might
be mapped to it.
-Notice that we represent each shredded field in `object` as a group of two
fields, `typed_value` and `variant_value`.
-We extract all homogenous data items of a certain path into `typed_value`, and
set aside incompatible data items in `variant_value`.
-Intuitively, incompatibilities within the same path may occur because we store
the shredding schema per Parquet file, and each file can contain several row
groups.
-Selecting a type for each field that is acceptable for all rows would be
impractical because it would require buffering the contents of an entire file
before writing.
+All fields for a variant, whether shredded or not, must be present in the
metadata.
-Typically, the expectation is that `variant_value` exists at every level as an
option, along with one of `object`, `array` or `typed_value`.
-If the actual Variant value contains a type that does not match the provided
schema, it is stored in `variant_value`.
-An `variant_value` may also be populated if an object can be partially
represented: any fields that are present in the schema must be written to those
fields, and any missing fields are written to `variant_value`.
+## Value Shredding
-The `metadata` column is unchanged from its unshredded representation, and may
be referenced in `variant_value` fields in the shredded data.
+Variant values are stored in Parquet fields named `variant_value`.
+Each `variant_value` field may have an associated shredded field named
`typed_value` that stores the value when it matches a specific type.
+For example, a Variant field, `measurement` may be shredded as long values by
adding `typed_value` with type `int64`:
```
-optional group variant_col {
- required binary metadata;
- optional binary variant_value;
- optional group object {
- optional group a {
- optional binary variant_value;
- optional int64 typed_value;
- }
- optional group b {
- optional binary variant_value;
- optional group object {
- optional group c {
- optional binary variant_value;
- optional binary typed_value (STRING);
+optional group measurement (VARIANT) {
+ required binary metadata;
+ optional binary variant_value;
+ optional int64 typed_value;
+}
+```
+
+Both `variant_value` and `typed_value` are optional fields used together to
encode a single value.
+Values in the two fields must be interpreted according to the following table:
+
+| `variant_value` | `typed_value` | Meaning |
+| null | null | The value is missing |
+| non-null | null | The value is present and may be any type,
including null |
+| null | non-null | The value is present and the shredded type
|
+| non-null | non-null | The value is present and a partially
shredded object |
+
+An object is _partially shredded_ when the `variant_value` is an object and
the `typed_value` is a shredded object.
+
+If both fields are non-null and either is not an object, the value is invalid.
Readers must either fail or return the `variant_value`.
+
+### Shredded Value Types
+
+Shredded values must use the following Parquet types:
+
+| Variant Type | Equivalent Parquet Type |
+|-----------------------------|------------------------------|
+| boolean | BOOLEAN |
+| int8 | INT(8, signed=true) |
+| int16 | INT(16, signed=true) |
+| int32 | INT32 / INT(32, signed=true) |
+| int64 | INT64 / INT(64, signed=true) |
+| float | FLOAT |
+| double | DOUBLE |
+| decimal4 | DECIMAL(precision, scale) |
+| decimal8 | DECIMAL(precision, scale) |
+| decimal16 | DECIMAL(precision, scale) |
+| date | DATE |
+| timestamp | TIMESTAMP(true, MICROS) |
+| timestamp without time zone | TIMESTAMP(false, MICROS) |
+| binary | BINARY |
+| string | STRING |
+| array | LIST; see Arrays below |
+| object | GROUP; see Objects below |
+
+#### Primitive Types
+
+Primitive values can be shredded using the equivalent Parquet primitive type
from the table above for `typed_object`.
+
+Unless the value is shredded in an object field, `typed_value` or
`variant_value` (but not both) must be non-null.
+
+#### Arrays
+
+Arrays can be shredded using a 3-level Parquet list for `typed_value`.
+
+If the value is not an array, `typed_value` must be null.
+If the value is an array, `variant_value` must be null.
+
+The list `element` must be a required group that contains a `variant_type`
(`binary`) and may contain a shredded `typed_value` field.
+
+For example, a `tags` Variant may be shredded as a list of strings using the
following definition:
+```
+optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary variant_value;
+ optional group typed_value (LIST) { // must be optional to allow a null list
+ repeated group list {
+ required group element {
+ optional binary variant_value;
+ optional binary typed_value (STRING);
+ }
}
- }
}
- }
}
```
-| Variant Value | Top-level variant_value | b.variant_value | a.typed_value |
a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes |
-|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------|
-| {a: 123, b: {c: “hello”}} | null | null | 123 | null | hello | null | All
values shredded |
-| {a: 1.23, b: {c: “123”}} | null | null | null | 1.23 | 123 | null | a is not
an integer |
-| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c
set to non-null to indicate VariantNull |
-| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to
null, to indicate that c is missing |
-| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field
d is stored as variant_value |
-| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c:
4}}] | null | null | null | null | null | Not an object |
+All elements of an array must be non-null, since `array` elements cannote be
missing.
+Either `typed_value` or `variant_value` (but not both) must be non-null.
+
+#### Objects (Option 1)
-# Parquet Layout
+Fields of an object can be shredded using a Parquet group for `typed_value`
that contains shredded fields.
-The `array` and `object` fields represent Variant array and object types,
respectively.
-Arrays must use the three-level list structure described in
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md.
+If the value is not an object, `typed_value` must be null.
-An `object` field must be a group.
-Each field name of this inner group corresponds to the Variant value's object
field name.
-Each inner field's type is a recursively shredded variant value: that is, the
fields of each object field must be one or more of `object`, `array`,
`typed_value` or `variant_value`.
+<!-- TODO: Why not keep non-shredded fields in the object itself? -->
+If the value is a partially shredded object, the `variant_value` must not
contain shredded fields. If such fields are present, the object is invalid and
readers must either fail or use the values from the `variant_value`.
-Similarly the elements of an `array` must be a group containing one or more of
`object`, `array`, `typed_value` or `variant_value`.
+Each shredded field is represented as a required group that contains a
`variant_value` and a `typed_value` field.
Review Comment:
The primary purpose is to reduce the number of cases that implementers have
to deal with. If all of the cases can be expressed with 2 optional fields
rather than 2 optional fields inside an optional group, then the group should
be required to simplify as much as possible.
In addition, every level in Parquet that is optional introduces another
repetition/definition level. That adds up quickly with nested structures and
ends up taking unnecessary space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
