I also favor moving forward with the proposal as is. I see strong value and use cases for the numeric vector type, and I agree with the restrictions around NaN/Inf values and element types in general. Precedent exists in other types that restrict which physical type values are logically valid (e.g., the geospatial types needing to be WKB-encoded). A general fixed-size list also has value, but I think it makes sense to separate the two types. The semantics are relevant, and I can see how the two types might diverge further in the future regarding statistics or encodings.
I would be okay with renaming the vector type, but it seems like the VECTOR name is already used across other engines mentioned in this thread. Best, Philipp On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > To separate orthogonal concerns, I feel we should treat the general > FixedSizeList as the underlying physical layout, and separate it from the > discussion of a specialized VECTOR (or broader ML tensor) logical type. > > Currently, the immediate need is to define a specialized VECTOR type so > that different engines and formats have a common semantic contract for > interoperability. > > If the main point of contention is just the NaN/Inf/null limitations, maybe > we can resolve this by adding a qualification to the name (like Andrew > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion with > broader ML use cases. Alternatively, we could introduce a boolean flag > (e.g., optional bool VectorType.finite) as proposed earlier. > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <[email protected]> > wrote: > > > I am a little confused about the points of disagreement here. > > > > Is it that the name `VECTOR` is too general for the specific use case of > > fixed length arrays with non-finite values used for vector embeddings? > If > > so, maybe we could add a qualification to the name (FINITE_VECTOR? > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > It seems like there is precedent for defining logical types that specify > > certain physical types and restrict their contents to help readers > > interpret meaning, for example string[1] or JSON; Defining one for > vector > > embeddings seems reasonable enough to me. > > > > Andrew > > > > > > [1]: > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> wrote: > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > Vector stores can choose whatever policy fits their usage, but we're > > > > talking about a general-purpose file format that aims to be broadly > > > > applicable. > > > > > > > > > I agree that the format is intended to be general purpose, but types > > within > > > the format define specific representation and semantics to achieve that > > > goal. > > > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > The justification for restricting non-finite values is directly > related > > to > > > the semantic meaning of a vector. Vectors have specific > characteristics > > > (dimension, magnitude, direction, etc). Non-finite values are invalid > > and > > > break those characteristics. I'd liken it to storing non-utf8 values > in > > a > > > string and still calling it a string. > > > > > > That also differentiates a vector from a fixed length list, which is > more > > > general and does not carry the same semantic meaning. > > > > > > I also don't think we need to bundle all similarly shaped use cases > into > > > the same solution (e.g. flxed size list). Vector is a well established > > > data type at this point and aligns with the general purpose nature of > the > > > format (many other formats and databases natively support this). > > > > > > -Dan > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <[email protected]> > > > wrote: > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > I think something I haven't seen yet is the clear use-case for > > > non-finite > > > > > values within a vector type. There has been a lot of theoretical > > > > discussion > > > > > here about use cases but what toolings currently produce these > > vectors > > > > and > > > > > how do they function in downstream engines? I'd like to limit > > > introducing > > > > > types that have a wide range of values which are simply are failure > > > modes > > > > > of different flavors > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or > > > > any other piece of dataframe or scientific computing software). > > > > > > > > It's not like the general concept of a vector is a IETF standard or > > > > anything, so analogies with JSON or UTF8 (where the values are > expected > > > > to be valid according to the respective specs) do not apply. > > > > > > > > This discussion would IMHO have a better chance of converging if we > > > > separated the two orthogonal concerns of: > > > > > > > > 1. representing fixed-length list/vector data in Parquet (with a > > > > potential limitation to non-nested types) > > > > > > > > 2. representing the information that a given column (or column chunk, > > > > etc.) does not contain non-finite values > > > > > > > > ... both of which are useful on their own right, but do not have to > be > > > > conflated and bundled together. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > >
