stevenzwu commented on code in PR #18416:
URL: https://github.com/apache/iceberg/pull/18416#discussion_r4546865464


##########
format/spec.md:
##########
@@ -1934,6 +1945,7 @@ Lists must use the [3-level 
representation](https://github.com/apache/parquet-fo
 | **`binary`**       | `binary`                                                
                                                                                
     |                                             |                            
                                    |
 | **`struct`**       | `group`                                                 
                                                                                
     |                                             |                            
                                    |
 | **`list`**         | `3-level list`                                          
                                                                                
     | `LIST`                                      | See Parquet docs for 
3-level representation.                   |
+| **`vector(E, D)`** | `3-level list`                                          
                                                                                
     | `LIST`                                      | The outer group must 
contain a single repeated group named `list`. The `list` group must contain a 
single primitive field named `element`, marked `required` in the Parquet 
schema. The `element` field must use the same Parquet physical type as an 
Iceberg column of type E, following the mapping in this table. Each non-null 
vector must contain exactly D non-null, finite elements. |

Review Comment:
   Parquet FINITE_VECTOR spec was merged. So this row can reference it now. 
   
   Parquet does not forbid field IDs on inner nodes; the rule here is an 
Iceberg one, because a vector is a primitive with a single column ID on the 
outer group, matching the Avro element-id rule.
   
   ```suggestion
   | **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group 
carries the field ID of the vector. The `element` field must use the Parquet 
physical type for E from this table. The ConvertedType must be `LIST`. The 
`list` and `element` fields must not be assigned field IDs. |
   ```



##########
format/spec.md:
##########
@@ -282,6 +283,7 @@ Supported primitive types are defined in the table below. 
Primitive types added
 |                  | **`uuid`**         | Universally unique identifiers       
                                    | Should use 16-byte fixed                  
       |
 |                  | **`fixed(L)`**     | Fixed-length byte array of length L  
                                    |                                           
       |
 |                  | **`binary`**       | Arbitrary-length byte array          
                                    |                                           
       |
+| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite 
numeric values with element type E and dimension D | E must be `int`, `long`, 
`float`, or `double`; D must be positive |

Review Comment:
   Ordering is needed in two places:
   - Sort fields: the Sorting section reuses the partition transform table, so 
`identity(vector)` would be a valid sort field, but the spec defines no order 
for vectors. Two writers could sort the same sort order differently.
   - Partition statistics file: rows must be sorted by `partition`, which 
includes identity values.
   
   Either define an order for vector, or exclude `vector` from `identity` 
(#18545), which removes both cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to