Hi everyone, I’d like to start a discussion thread on adding first-class vector type support to Iceberg.
Vector embeddings are increasingly common in AI/ML workloads. Today they can be stored as *list<float>*, but that does not capture the key invariant that every non-null value has the same dimension. There is also active work <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0> in the Parquet community around fixed-size vector/list support, and this proposal focuses on the Iceberg table-format layer. The preferred direction described in the proposal is a dense numeric vector type with compact schema encoding such as *float[768]* or *int[128]*: fixed dimension, numeric coordinates, non-null elements, and top-level nullability controlled by the field’s required flag. The proposal also discusses things like file-format mapping and schema evolution, while questions around metrics, constraint, and future extensions remain open. Link to the proposal: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE Looking forward to the community feedback. Thank you, Yan
