RussellSpitzer commented on code in PR #18416: URL: https://github.com/apache/iceberg/pull/18416#discussion_r4557115865
########## format/spec.md: ########## @@ -1934,6 +1945,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | +| **`vector(E, D)`** | `3-level list` | `LIST` | The outer group must contain a single repeated group named `list`. The `list` group must contain a single primitive field named `element`, marked `required` in the Parquet schema. The `element` field must use the same Parquet physical type as an Iceberg column of type E, following the mapping in this table. Each non-null vector must contain exactly D non-null, finite elements. | Review Comment: I would agree on the "must" unless we want to do possible up/down casting which I'm beginning to think would just encourage type promotion. I think it's probably simpler (if we want to include float16) to treat vector elements as seperate from the Iceberg Type System. I would just detach the element completely from the Iceberg Type and define it them in their own little table. ``` | **`vector(E, D)`** | `group` | `FINITE_VECTOR(D)` | The `element` field must use the Parquet type for E listed in the vector element table below. See Parquet docs for [`FINITE_VECTOR`](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#finite-vectors). | ``` ``` Vector elements are stored in Parquet using the following types: | Element type | Parquet physical type | Logical type | |---------------|---------------------------|--------------| | **`int`** | `int32` | | | **`long`** | `int64` | | | **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16` | | **`float`** | `float` | | | **`double`** | `double` | | ``` We Could do something similar in the Iceberg Vector Definitions above ``` | [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of D finite numeric values with element type E | E must be a [vector element type](#vector); D must be positive | ``` ``` ##### Vector A `vector(E, D)` contains exactly D elements of element type E, and every element is non-null. The vector value may be null only when the vector field is optional. E must be one of the following vector element types: | Element type | Element values | Value type | |---------------|------------------------------------------------------------------------|------------| | **`int`** | 32-bit signed integers | `int` | | **`long`** | 64-bit signed integers | `long` | | **`float16`** | `float` values that are exactly representable as IEEE 754 binary16 | `float` | | **`float`** | 32-bit IEEE 754 floating point | `float` | | **`double`** | 64-bit IEEE 754 floating point | `double` | Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including equality and JSON single-value serialization. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even if every list instance contains exactly D elements. Equality of vector values is element-wise, using the equality rules of the value type. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
