PFischbeck commented on code in PR #18416:
URL: https://github.com/apache/iceberg/pull/18416#discussion_r4551680640
##########
format/spec.md:
##########
@@ -282,6 +283,7 @@ Supported primitive types are defined in the table below.
Primitive types added
| | **`uuid`** | Universally unique identifiers
| Should use 16-byte fixed
|
| | **`fixed(L)`** | Fixed-length byte array of length L
|
|
| | **`binary`** | Arbitrary-length byte array
|
|
+| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite
numeric values with element type E and dimension D | E must be `int`, `long`,
`float`, or `double`; D must be positive |
Review Comment:
Agreed, I think we should exclude `vector` from `identity`, and likely also
from constraints.
##########
format/spec.md:
##########
@@ -1934,6 +1945,7 @@ Lists must use the [3-level
representation](https://github.com/apache/parquet-fo
| **`binary`** | `binary`
| |
|
| **`struct`** | `group`
| |
|
| **`list`** | `3-level list`
| `LIST` | See Parquet docs for
3-level representation. |
+| **`vector(E, D)`** | `3-level list`
| `LIST` | The outer group must
contain a single repeated group named `list`. The `list` group must contain a
single primitive field named `element`, marked `required` in the Parquet
schema. The `element` field must use the same Parquet physical type as an
Iceberg column of type E, following the mapping in this table. Each non-null
vector must contain exactly D non-null, finite elements. |
Review Comment:
> I don't like the wording "must" in `The element field must use`. This
makes FLOAT16 Parquet storage invalid and the Vector type useless for our
use-cases requiring performance.
@pvary I'm not sure I follow. The spec has to specify the Parquet types for,
e.g., Iceberg's `vector(float, 768)`, otherwise we are underspecified. Hence,
default option: the mapping comes from the native types in this table.
Your ask for FLOAT16, in my opinion, would still require explicit mention:
Can I use it for `vector(float)`? For `vector(double)`? What about smaller int
types? Either way, my understanding is that FLOAT16 support is still under
discussion.
##########
format/spec.md:
##########
@@ -297,6 +298,7 @@ Supported primitive types are defined in the table below.
Primitive types added
| | **`binary`** | Arbitrary-length byte array
|
|
| [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC –
Simple feature access][1001]. Edge-interpolation is always linear/planar. See
[Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not
specified, C is `OGC:CRS84`. |
|
| [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC –
Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes).
Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C
is `OGC:CRS84` and A is `spherical`. |
+| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite
numeric values with element type E and dimension D | E must be `int`, `long`,
`float`, or `double`; D must be positive |
Review Comment:
I don't think we should - Parquet has no such limit, and in Iceberg,
comparable types like `fixed(L)` also have no upper limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]