Thanks Yan, this is a good proposal. I agree with keeping the initial type narrow: numeric coordinates, fixed dimension, required elements, and nullability only at the vector-field level. That gives engines a useful semantic contract without pulling vector functions, indexes, or broader fixed-shape containers into the initial scope.
For the Parquet mapping, I think we should make the field-ID contract explicit. Since vector is modeled as a primitive semantic type in Iceberg, I would place the Iceberg field ID only on the outer Parquet VECTOR group. The intermediate vector node and element leaf should remain physical-mapping details rather than independently addressable Iceberg fields. The mapping should require the element leaf to be physically required, validate that its numeric type matches the Iceberg element type, and require the physical vector length to match the declared dimension. That keeps the compact float[768] representation consistent with the proposed evolution model. The metrics mapping also needs an explicit rule. Under the current Parquet VECTOR direction, the coordinate values live in a real leaf column, so Parquet’s per-column statistics value counts, null counts, min/max are element-level, whereas an Iceberg vector column has one logical value per row. We should not publish that leaf’s counts or bounds directly under the vector field ID. Consistent with Section 9.1, I would define value_counts and null_value_counts using vector-value semantics and initially omit bounds and NaN metrics unless coordinate-level metric semantics are standardized separately. For unsupported file formats, it may also be useful to distinguish historical files from new writes. Adding an optional vector column should not invalidate existing Avro or ORC files that predate the column; those files can project null for it. The restriction should apply to new files that materialize the vector column: those writes must use the supported Parquet representation and should fail during schema or write validation otherwise. Finally, the spec change should assign a minimum Iceberg format version. The logical type work can move forward while the Parquet representation is finalized, but write support should be enabled only once the physical contract is stable. Overall, I think the proposal is well scoped. The main areas I would make normative are the field-ID mapping, vector-level metric semantics, and behavior for existing versus newly written non-Parquet files. Regards, Tanmay Rauth On Fri, Jul 10, 2026 at 10:25 PM Yan Yan <[email protected]> wrote: > > Hi everyone, > > I’d like to start a discussion thread on adding first-class vector type support to Iceberg. > > Vector embeddings are increasingly common in AI/ML workloads. Today they can be stored as list<float>, but that does not capture the key invariant that every non-null value has the same dimension. There is also active work in the Parquet community around fixed-size vector/list support, and this proposal focuses on the Iceberg table-format layer. > > The preferred direction described in the proposal is a dense numeric vector type with compact schema encoding such as float[768] or int[128]: fixed dimension, numeric coordinates, non-null elements, and top-level nullability controlled by the field’s required flag. The proposal also discusses things like file-format mapping and schema evolution, while questions around metrics, constraint, and future extensions remain open. > > Link to the proposal: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE > > Looking forward to the community feedback. > > Thank you, > Yan >
