Thinking about this more, I am also fine with pulling out the “NaN/Inf” requirement. Either into a type boolean (my preference), or putting no restrictions on the Parquet level, allowing all IEEE 754 float numbers. If some top-level application wants to put additional constraints on the data type, like Iceberg not allowing NaN/infinity, it can still do so one layer above. It seems like this was the biggest point of discussion remaining. All these options work for me, and we should move forward with this proposal.
Concerning additional element types like bf16, they can be introduced as Parquet types (at least in the current LIST-based spec proposal). Concerning Will’s suggestion around representing VECTOR as a variable length list with a MIN/MAX_LENGTH metadata restriction: I think that a dedicated logical VECTOR type still provides enough benefits by describing the semantics of the data, and allowing to capture future changes around statistics. In that case, the length is better captured as an exact length for this type specifically. I’m also not sure about other use cases for MIN_LENGTH, so introducing a new type constraint system just for the MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my view. Best, Philipp On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <[email protected]> wrote: > +1 Sebastian, a vector type suggests usage beyond normalized embeddings and > engines really aren't going to be able to elide all the checks anyway. > > Antoine, you suggested we explore metadata solutions too :) > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in > Parquet. Engines can write opaque blobs they understand in the metadata > (e.g. Arrow does) so the extra type info round-trips, but this isn't > standardized, portable, nor interoperable. > > So perhaps standardizing constraint metadata or new Thrift fields (my pref) > can generically support VECTOR(N) whilst also being more broadly useful for > other use cases and data types? > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH for > Parquet's variable-length physcial types. These would count logical units > (e.g. UTF8 code points for STRING) rather than bytes. > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with > MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - can > pick up these constraints if they support doing so. We don't actually need > a logical type to express a VECTOR then. > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on > UTF8 columns etc. > > This also seems generally useful in the future as we approach expressing > other tensor data shapes and types. > > This won't change the mechanics: the storage type doesn't enforce these > constraints and engines must check them. It's previously established that > performance is just a software problem and there is no fundamental > performance advantage to the fixed size types in Dremel-like formats like > Parquet. > > Best, > Will > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <[email protected] > > > wrote: > > > After listening in on the last meeting Wednesday, I have been thinking > > about this quite a bit, and thought I would share my 2 cents. > > > > I suggest not requiring VECTOR elements to be finite, as proposed in the > PR > > <https://github.com/apache/parquet-format/pull/624>. > > > > While Divjot and others note the specialization for UTF-8, Parquet > already > > defines DOUBLE and FLOAT as IEEE encodings > > <https://github.com/apache/parquet-format/blob/master/README.md#types>. > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on > > those same underlying physical types. Scalar FLOAT/DOUBLE columns have > > never had such a rule. > > > > It would also not match how the systems that actually produce these > vectors > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy, > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. > > > > pgvector's "all elements must be finite" rule is an insert policy. While > I > > think Parquet as a general-purpose format should allow them out of the > box. > > It is a user system decision to disallow writing these values or to throw > > errors when encountering them. > > > > A type-level ban also does not keep Inf/NaN out of vector math. Finite > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32. > > > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR > itself > > was required to be finite. The ban only forbids storing specials in the > > coordinates, and does not prevent even simple downstream computations > from > > producing them. > > > > I suggest keeping VECTOR as the type and add an optional bool > > VectorType.finite (default false). Engines that want pgvector-like > > rejection can require the flag = true. Engines that use NumPy / PyTorch / > > Spark arrays can keep representable IEEE values. For the same reason, I > do > > not favor adding yet another type called FINITE_VECTOR. > > > > Best, > > Sebastian > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > [email protected]> > > wrote: > > > > > I agree with moving forward with the proposal and renaming VECTOR to > > > FINITE_VECTOR or something > > > similar if others feel that better communicates its intent. I see value > > in > > > having a vector type that guarantees > > > its values are mathematically-correct vectors for a few reasons: > > > > > > 1. As others have mentioned, this is similar to the STRING logical type > > > mandating UTF-8. > > > 2. It is not possible to rely on statistics for detecting NaN or +-inf > > > values as all statistics are optional. > > > To my knowledge, there is no precedent for requiring that writers emit > > > stats for specific logical types. > > > 3. Without statistics, NaN/inf values could flow through the engine and > > > block them for specific functionality. > > > This seems to be Lance's behavior. However, I don't think this is as > easy > > > as it sounds and Lance has run > > > into several bugs due to this ([1] [2] for examples). Their Python [3] > > and > > > Rust [4] APIs default to erroring on > > > vectors that contain NaN values. > > > > > > I agree that Parquet is a general-purpose format, but I don't think > this > > > implies the lack of domain-specific types. > > > Instead, this broadens the use cases where Parquet would be considered > a > > > good fit. I also fully agree with > > > continuing work on a general fixed-size list type if we believe there > are > > > non-vector use cases that would benefit > > > from such guarantees at the storage layer. > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in > > > "root > > > cause") > > > [3] > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > [4] > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > > [email protected]> wrote: > > > > > > > I also favor moving forward with the proposal as is. I see strong > value > > > and > > > > use cases for the numeric vector type, and I agree with the > > restrictions > > > > around NaN/Inf values and element types in general. Precedent exists > in > > > > other types that restrict which physical type values are logically > > valid > > > > (e.g., the geospatial types needing to be WKB-encoded). A general > > > > fixed-size list also has value, but I think it makes sense to > separate > > > the > > > > two types. The semantics are relevant, and I can see how the two > types > > > > might diverge further in the future regarding statistics or > encodings. > > > > > > > > I would be okay with renaming the vector type, but it seems like the > > > VECTOR > > > > name is already used across other engines mentioned in this thread. > > > > > > > > Best, > > > > Philipp > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > > > > > > > > > To separate orthogonal concerns, I feel we should treat the general > > > > > FixedSizeList as the underlying physical layout, and separate it > from > > > the > > > > > discussion of a specialized VECTOR (or broader ML tensor) logical > > type. > > > > > > > > > > Currently, the immediate need is to define a specialized VECTOR > type > > so > > > > > that different engines and formats have a common semantic contract > > for > > > > > interoperability. > > > > > > > > > > If the main point of contention is just the NaN/Inf/null > limitations, > > > > maybe > > > > > we can resolve this by adding a qualification to the name (like > > Andrew > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid > > confusion > > > > with > > > > > broader ML use cases. Alternatively, we could introduce a boolean > > flag > > > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > [email protected]> > > > > > wrote: > > > > > > > > > > > I am a little confused about the points of disagreement here. > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the specific use > > case > > > > of > > > > > > fixed length arrays with non-finite values used for vector > > > embeddings? > > > > > If > > > > > > so, maybe we could add a qualification to the name > (FINITE_VECTOR? > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > It seems like there is precedent for defining logical types that > > > > specify > > > > > > certain physical types and restrict their contents to help > readers > > > > > > interpret meaning, for example string[1] or JSON; Defining one > for > > > > > vector > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits their usage, > but > > > > we're > > > > > > > > talking about a general-purpose file format that aims to be > > > broadly > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general purpose, but > > > types > > > > > > within > > > > > > > the format define specific representation and semantics to > > achieve > > > > that > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > trivial > > > to > > > > > > > > produce vectors with non-finite values using e.g. NumPy or > > Pandas > > > > (or > > > > > > > > any other piece of dataframe or scientific computing > software). > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite values is > directly > > > > > related > > > > > > to > > > > > > > the semantic meaning of a vector. Vectors have specific > > > > > characteristics > > > > > > > (dimension, magnitude, direction, etc). Non-finite values are > > > > invalid > > > > > > and > > > > > > > break those characteristics. I'd liken it to storing non-utf8 > > > values > > > > > in > > > > > > a > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > That also differentiates a vector from a fixed length list, > which > > > is > > > > > more > > > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > > > > > I also don't think we need to bundle all similarly shaped use > > cases > > > > > into > > > > > > > the same solution (e.g. flxed size list). Vector is a well > > > > established > > > > > > > data type at this point and aligns with the general purpose > > nature > > > of > > > > > the > > > > > > > format (many other formats and databases natively support > this). > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > > I think something I haven't seen yet is the clear use-case > > for > > > > > > > non-finite > > > > > > > > > values within a vector type. There has been a lot of > > > theoretical > > > > > > > > discussion > > > > > > > > > here about use cases but what toolings currently produce > > these > > > > > > vectors > > > > > > > > and > > > > > > > > > how do they function in downstream engines? I'd like to > limit > > > > > > > introducing > > > > > > > > > types that have a wide range of values which are simply are > > > > failure > > > > > > > modes > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > trivial > > > to > > > > > > > > produce vectors with non-finite values using e.g. NumPy or > > Pandas > > > > (or > > > > > > > > any other piece of dataframe or scientific computing > software). > > > > > > > > > > > > > > > > It's not like the general concept of a vector is a IETF > > standard > > > or > > > > > > > > anything, so analogies with JSON or UTF8 (where the values > are > > > > > expected > > > > > > > > to be valid according to the respective specs) do not apply. > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance of converging > > if > > > we > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet > (with > > a > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > 2. representing the information that a given column (or > column > > > > chunk, > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > ... both of which are useful on their own right, but do not > > have > > > to > > > > > be > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
