stevenzwu commented on code in PR #18416: URL: https://github.com/apache/iceberg/pull/18416#discussion_r4546865464
########## format/spec.md: ########## @@ -1934,6 +1945,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | +| **`vector(E, D)`** | `3-level list` | `LIST` | The outer group must contain a single repeated group named `list`. The `list` group must contain a single primitive field named `element`, marked `required` in the Parquet schema. The `element` field must use the same Parquet physical type as an Iceberg column of type E, following the mapping in this table. Each non-null vector must contain exactly D non-null, finite elements. | Review Comment: Parquet FINITE_VECTOR spec was merged. So this row can reference it now. Parquet does not forbid field IDs on inner nodes; the rule here is an Iceberg one, because a vector is a primitive with a single column ID on the outer group, matching the Avro element-id rule. ```suggestion | **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. The `element` field must use the Parquet physical type for E from this table. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | ``` ########## format/spec.md: ########## @@ -282,6 +283,7 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`** | Universally unique identifiers | Should use 16-byte fixed | | | **`fixed(L)`** | Fixed-length byte array of length L | | | | **`binary`** | Arbitrary-length byte array | | +| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite numeric values with element type E and dimension D | E must be `int`, `long`, `float`, or `double`; D must be positive | Review Comment: Ordering is needed in two places: - Sort fields: the Sorting section reuses the partition transform table, so `identity(vector)` would be a valid sort field, but the spec defines no order for vectors. Two writers could sort the same sort order differently. - Partition statistics file: rows must be sorted by `partition`, which includes identity values. Either define an order for vector, or exclude `vector` from `identity` (#18545), which removes both cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
