To separate orthogonal concerns, I feel we should treat the general FixedSizeList as the underlying physical layout, and separate it from the discussion of a specialized VECTOR (or broader ML tensor) logical type.
Currently, the immediate need is to define a specialized VECTOR type so that different engines and formats have a common semantic contract for interoperability. If the main point of contention is just the NaN/Inf/null limitations, maybe we can resolve this by adding a qualification to the name (like Andrew suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion with broader ML use cases. Alternatively, we could introduce a boolean flag (e.g., optional bool VectorType.finite) as proposed earlier. On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <[email protected]> wrote: > I am a little confused about the points of disagreement here. > > Is it that the name `VECTOR` is too general for the specific use case of > fixed length arrays with non-finite values used for vector embeddings? If > so, maybe we could add a qualification to the name (FINITE_VECTOR? > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > It seems like there is precedent for defining logical types that specify > certain physical types and restrict their contents to help readers > interpret meaning, for example string[1] or JSON; Defining one for vector > embeddings seems reasonable enough to me. > > Andrew > > > [1]: > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> wrote: > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > Vector stores can choose whatever policy fits their usage, but we're > > > talking about a general-purpose file format that aims to be broadly > > > applicable. > > > > > > I agree that the format is intended to be general purpose, but types > within > > the format define specific representation and semantics to achieve that > > goal. > > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > > any other piece of dataframe or scientific computing software). > > > > > > The justification for restricting non-finite values is directly related > to > > the semantic meaning of a vector. Vectors have specific characteristics > > (dimension, magnitude, direction, etc). Non-finite values are invalid > and > > break those characteristics. I'd liken it to storing non-utf8 values in > a > > string and still calling it a string. > > > > That also differentiates a vector from a fixed length list, which is more > > general and does not carry the same semantic meaning. > > > > I also don't think we need to bundle all similarly shaped use cases into > > the same solution (e.g. flxed size list). Vector is a well established > > data type at this point and aligns with the general purpose nature of the > > format (many other formats and databases natively support this). > > > > -Dan > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <[email protected]> > > wrote: > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > I think something I haven't seen yet is the clear use-case for > > non-finite > > > > values within a vector type. There has been a lot of theoretical > > > discussion > > > > here about use cases but what toolings currently produce these > vectors > > > and > > > > how do they function in downstream engines? I'd like to limit > > introducing > > > > types that have a wide range of values which are simply are failure > > modes > > > > of different flavors > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > > any other piece of dataframe or scientific computing software). > > > > > > It's not like the general concept of a vector is a IETF standard or > > > anything, so analogies with JSON or UTF8 (where the values are expected > > > to be valid according to the respective specs) do not apply. > > > > > > This discussion would IMHO have a better chance of converging if we > > > separated the two orthogonal concerns of: > > > > > > 1. representing fixed-length list/vector data in Parquet (with a > > > potential limitation to non-nested types) > > > > > > 2. representing the information that a given column (or column chunk, > > > etc.) does not contain non-finite values > > > > > > ... both of which are useful on their own right, but do not have to be > > > conflated and bundled together. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > >
