wgtmac commented on code in PR #624:
URL: https://github.com/apache/parquet-format/pull/624#discussion_r4432015891
##########
LogicalTypes.md:
##########
@@ -1097,6 +1097,77 @@ optional group my_map (MAP_KEY_VALUE) {
}
```
+### Vectors
+
+`VECTOR` is used to annotate fixed-length ordered sequences of finite and
+non-null elements.
+
+`VectorType` annotation has one required parameter, `num_elements`, which is
+the number of elements in each non-null vector and MUST be greater than zero.
+
+`VECTOR` MUST always annotate the canonical 3-level structure:
+
+```
+<vector-repetition> group <name> (VECTOR(<num_elements>)) {
+ repeated group list {
+ required <element-type> element;
+ }
+}
+```
+
+* The outer level must be a group with `logicalType` set to `VECTOR` and
+ `converted_type` set to `LIST`. Its repetition must either be `optional` or
Review Comment:
`converted_type` has been deprecated so it might be better not to mention it?
##########
LogicalTypes.md:
##########
@@ -1097,6 +1097,77 @@ optional group my_map (MAP_KEY_VALUE) {
}
```
+### Vectors
+
+`VECTOR` is used to annotate fixed-length ordered sequences of finite and
+non-null elements.
+
+`VectorType` annotation has one required parameter, `num_elements`, which is
+the number of elements in each non-null vector and MUST be greater than zero.
+
+`VECTOR` MUST always annotate the canonical 3-level structure:
+
+```
+<vector-repetition> group <name> (VECTOR(<num_elements>)) {
+ repeated group list {
+ required <element-type> element;
+ }
+}
+```
+
+* The outer level must be a group with `logicalType` set to `VECTOR` and
+ `converted_type` set to `LIST`. Its repetition must either be `optional` or
+ `required` and determines whether the vector may be null. It must contain a
+ single field named `list`.
+* The middle level must be a repeated group named `list` with a single field
+ named `element`.
+* The element MUST be a `required` primitive field, individual elements MUST
+ NOT be null.
+
+The element types supported are:
+
+* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
+* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
+ `DECIMAL`
+* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.
+
+The 2-level structures accepted for `LIST` under its backward-compatibility
Review Comment:
This seems redundant. But if we want to make it explicit, we need to exclude
the one-level repeated primitive node as well.
##########
src/main/thrift/parquet.thrift:
##########
@@ -479,6 +479,18 @@ struct GeographyType {
struct FileType {
}
+/**
+* Fixed-length vector logical type annotation
+*
+* Annotates the outer group of a three-level LIST structure. Each
Review Comment:
Not a native speaker but I don't think it is `a three-level LIST structure`.
Perhaps better to say `same as the LIST-annotated three-level structure`?
##########
LogicalTypes.md:
##########
@@ -1097,6 +1097,77 @@ optional group my_map (MAP_KEY_VALUE) {
}
```
+### Vectors
+
+`VECTOR` is used to annotate fixed-length ordered sequences of finite and
+non-null elements.
+
+`VectorType` annotation has one required parameter, `num_elements`, which is
+the number of elements in each non-null vector and MUST be greater than zero.
+
+`VECTOR` MUST always annotate the canonical 3-level structure:
+
+```
+<vector-repetition> group <name> (VECTOR(<num_elements>)) {
+ repeated group list {
+ required <element-type> element;
+ }
+}
+```
+
+* The outer level must be a group with `logicalType` set to `VECTOR` and
+ `converted_type` set to `LIST`. Its repetition must either be `optional` or
+ `required` and determines whether the vector may be null. It must contain a
+ single field named `list`.
+* The middle level must be a repeated group named `list` with a single field
+ named `element`.
+* The element MUST be a `required` primitive field, individual elements MUST
+ NOT be null.
+
+The element types supported are:
+
+* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
+* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
+ `DECIMAL`
+* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.
+
+The 2-level structures accepted for `LIST` under its backward-compatibility
+rules are not valid for `VECTOR`.
+
+Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`,
+this excludes NaN, positive infinity, and negative infinity.
+
+For example, a nullable vector containing 768 required `FLOAT` elements is:
+
+```
+optional group embedding (VECTOR(768)) {
+ repeated group list {
+ required float element;
+ }
+}
+```
+
+Every non-null vector must contain exactly `num_elements` elements.
+Because `num_elements` is greater than zero, a non-null vector cannot be empty.
+Writers MUST enforce element count, non-null-element, and finite-element
+requirements. Readers MAY rely on these requirements without validating them. A
+reader that detects a different element count, a null element, or a non-finite
+element MUST treat the data as invalid.
+
+For example, with `num_elements = 3` and a `required float element`,
+`[1.0, 2.0, 3.0]` is valid. `[]`, `[1.0, 2.0]`, `[1.0, 2.0, 3.0, 4.0]`,
+`[1.0, null, 3.0]`, `[null, null, null]`, `[1.0, NaN, 3.0]`, and
+`[1.0, +Infinity, 3.0]` are invalid. A null vector is valid only when the
+outer group is `optional`.
+
+The sort order of `VECTOR` values is undefined and no statistics are defined
+for whole vectors.
+
+`VECTOR` corresponds to the `LIST` ConvertedType. Writers must set both
Review Comment:
Ah, now I know the intention for mentioning the converted type. But this
still looks partial because legacy readers that expect a `LIST` logical type
but ignore converted type may interpret the vector type as a struct type of a
single repeated one-element struct or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]