I am a little confused about the points of disagreement here. Is it that the name `VECTOR` is too general for the specific use case of fixed length arrays with non-finite values used for vector embeddings? If so, maybe we could add a qualification to the name (FINITE_VECTOR? FEATURE_VECTOR? VECTOR_EMBEDDING?)
It seems like there is precedent for defining logical types that specify certain physical types and restrict their contents to help readers interpret meaning, for example string[1] or JSON; Defining one for vector embeddings seems reasonable enough to me. Andrew [1]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> wrote: > > > > Well, Parquet is not a single-purpose format. > > > > Vector stores can choose whatever policy fits their usage, but we're > > talking about a general-purpose file format that aims to be broadly > > applicable. > > > I agree that the format is intended to be general purpose, but types within > the format define specific representation and semantics to achieve that > goal. > > > > "Vector" has a more general sense than ML and it's quite trivial to > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > any other piece of dataframe or scientific computing software). > > > The justification for restricting non-finite values is directly related to > the semantic meaning of a vector. Vectors have specific characteristics > (dimension, magnitude, direction, etc). Non-finite values are invalid and > break those characteristics. I'd liken it to storing non-utf8 values in a > string and still calling it a string. > > That also differentiates a vector from a fixed length list, which is more > general and does not carry the same semantic meaning. > > I also don't think we need to bundle all similarly shaped use cases into > the same solution (e.g. flxed size list). Vector is a well established > data type at this point and aligns with the general purpose nature of the > format (many other formats and databases natively support this). > > -Dan > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <[email protected]> > wrote: > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > I think something I haven't seen yet is the clear use-case for > non-finite > > > values within a vector type. There has been a lot of theoretical > > discussion > > > here about use cases but what toolings currently produce these vectors > > and > > > how do they function in downstream engines? I'd like to limit > introducing > > > types that have a wide range of values which are simply are failure > modes > > > of different flavors > > > > "Vector" has a more general sense than ML and it's quite trivial to > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > any other piece of dataframe or scientific computing software). > > > > It's not like the general concept of a vector is a IETF standard or > > anything, so analogies with JSON or UTF8 (where the values are expected > > to be valid according to the respective specs) do not apply. > > > > This discussion would IMHO have a better chance of converging if we > > separated the two orthogonal concerns of: > > > > 1. representing fixed-length list/vector data in Parquet (with a > > potential limitation to non-nested types) > > > > 2. representing the information that a given column (or column chunk, > > etc.) does not contain non-finite values > > > > ... both of which are useful on their own right, but do not have to be > > conflated and bundled together. > > > > Regards > > > > Antoine. > > > > > > >
