Hi Yan, thanks for the proposal! I like it and agree with focusing on a dense numeric vector type that contains non-null elements. This matches the semantics and scope of existing types in various database systems. The ongoing Parquet fixed-size list/vector should be the basis to utilise efficient storage and reduce the shape constraints that the Iceberg engine must enforce.
I suggest keeping the feature list lean, meaning no additional vector-specific stats, constraints, or schema evolution. I agree with Tanmay that we should define metrics (value count, null count) based on the vector level instead of the element level, and defer other bounds/metrics for now. Such bounds and additional stats might also make sense at the Parquet level and could grow from there in a follow-up. Best Philipp On Sat, Jul 11, 2026 at 8:34 PM Tanmay Rauth <[email protected]> wrote: > > Thanks Yan, this is a good proposal. I agree with keeping the initial type > narrow: numeric coordinates, fixed dimension, required elements, and > nullability only at the vector-field level. That gives engines a useful > semantic contract without pulling vector functions, indexes, or broader > fixed-shape containers into the initial scope. > > For the Parquet mapping, I think we should make the field-ID contract > explicit. Since vector is modeled as a primitive semantic type in Iceberg, I > would place the Iceberg field ID only on the outer Parquet VECTOR group. The > intermediate vector node and element leaf should remain physical-mapping > details rather than independently addressable Iceberg fields. The mapping > should require the element leaf to be physically required, validate that its > numeric type matches the Iceberg element type, and require the physical > vector length to match the declared dimension. That keeps the compact > float[768] representation consistent with the proposed evolution model. > > The metrics mapping also needs an explicit rule. Under the current Parquet > VECTOR direction, the coordinate values live in a real leaf column, so > Parquet’s per-column statistics value counts, null counts, min/max are > element-level, whereas an Iceberg vector column has one logical value per > row. We should not publish that leaf’s counts or bounds directly under the > vector field ID. Consistent with Section 9.1, I would define value_counts and > null_value_counts using vector-value semantics and initially omit bounds and > NaN metrics unless coordinate-level metric semantics are standardized > separately. > > For unsupported file formats, it may also be useful to distinguish historical > files from new writes. Adding an optional vector column should not invalidate > existing Avro or ORC files that predate the column; those files can project > null for it. The restriction should apply to new files that materialize the > vector column: those writes must use the supported Parquet representation and > should fail during schema or write validation otherwise. > > Finally, the spec change should assign a minimum Iceberg format version. The > logical type work can move forward while the Parquet representation is > finalized, but write support should be enabled only once the physical > contract is stable. > > Overall, I think the proposal is well scoped. The main areas I would make > normative are the field-ID mapping, vector-level metric semantics, and > behavior for existing versus newly written non-Parquet files. > > > Regards, > Tanmay Rauth > > > > On Fri, Jul 10, 2026 at 10:25 PM Yan Yan <[email protected]> wrote: > > > > Hi everyone, > > > > I’d like to start a discussion thread on adding first-class vector type > > support to Iceberg. > > > > Vector embeddings are increasingly common in AI/ML workloads. Today they > > can be stored as list<float>, but that does not capture the key invariant > > that every non-null value has the same dimension. There is also active work > > in the Parquet community around fixed-size vector/list support, and this > > proposal focuses on the Iceberg table-format layer. > > > > The preferred direction described in the proposal is a dense numeric vector > > type with compact schema encoding such as float[768] or int[128]: fixed > > dimension, numeric coordinates, non-null elements, and top-level > > nullability controlled by the field’s required flag. The proposal also > > discusses things like file-format mapping and schema evolution, while > > questions around metrics, constraint, and future extensions remain open. > > > > Link to the proposal: > > https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE > > > > Looking forward to the community feedback. > > > > Thank you, > > Yan
