emkornfield commented on code in PR #456: URL: https://github.com/apache/parquet-format/pull/456#discussion_r1854919471
########## VariantEncoding.md: ########## @@ -0,0 +1,429 @@ +<!-- + - Licensed to the Apache Software Foundation (ASF) under one + - or more contributor license agreements. See the NOTICE file + - distributed with this work for additional information + - regarding copyright ownership. The ASF licenses this file + - to you under the Apache License, Version 2.0 (the + - "License"); you may not use this file except in compliance + - with the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, + - software distributed under the License is distributed on an + - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + - KIND, either express or implied. See the License for the + - specific language governing permissions and limitations + - under the License. + --> + +# Variant Binary Encoding + +> [!IMPORTANT] +> **This specification is still under active development, and has not been formally adopted.** + +A Variant represents a type that contain one of: +- Primitive: A type and corresponding value (e.g. INT, STRING) +- Array: An ordered list of Variant values +- Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys. + +A Variant is encoded with 2 binary values, the [value](#value-encoding) and the [metadata](#metadata-encoding). + +There are a fixed number of allowed primitive types, provided in the table below. +These represent a commonly supported subset of the [logical types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) allowed by the Parquet format. + +The Variant Binary Encoding allows representation of semi-structured data (e.g. JSON) in a form that can be efficiently queried by path. +The design is intended to allow efficient access to nested data even in the presence of very wide or deep structures. + +Another motivation for the representation is that (aside from metadata) each nested Variant value is contiguous and self-contained. +For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant. + +This document describes the Variant Binary Encoding scheme. +[VariantShredding.md](VariantShredding.md) describes the details of the Variant shredding scheme. + +# Variant in Parquet +A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`. +Both fields `value` and `metadata` are of type `binary`, and cannot be `null`. + +# Metadata encoding + +The encoded metadata always starts with a header byte. +``` + 7 6 5 4 3 0 + +-------+---+---+---------------+ +header | | | | version | Review Comment: it looks like bit 5 is unused? can we specify that is should always be zero? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
