I also think a FINITE_VECTOR type brings a lot of values to many use cases. It is would be great to make progress on the FINITE_VECTOR, and not to fit every use cases into the FIXED_LENGTH_LIST.
On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> wrote: > Going through this thread again, it looks like there are a few things that > people generally agree on: > > - In the short term, we want to introduce a logical type or types; a better > physical representation can be done later. We can focus on logical types. > - There is precedent in logical types for restricting values, like UTF8 > strings to contain only UTF-8 codepoints. I think we also agree that we > could restrict vector elements to non-nested or primitive types. > - There are many examples of databases that restrict their vector types to > non-null, finite elements. > - Some people want to have write-time guarantees that vectors have only > non-null, finite elements > - Some people want to extend this work to include tensors, shorter floating > point representations, variable lengths, NaN or Infinity elements, etc. > > It sounds like we are nearly there, but have two halves of the community > that care about different things. That just means we should have multiple > types. It's entirely reasonable to want a type that allows NaN values, and > it's also reasonable for those of us building databases, table standards, > and vector indexes to care about guarantees that elements are finite. > There's more than enough examples to show that this is a legitimate use > case. > > Since I don't see the concerns of the different use cases changing, it > looks like we need to find a compromise to move forward. I propose that we > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or > another VECTOR type that allows infinite and NaN values. > > Can we agree on a compromise? Is there a strong reason to try to fit these > use cases into a single type? What if that means not making progress? > > Ryan > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < > [email protected]> wrote: > > > Thinking about this more, I am also fine with pulling out the “NaN/Inf” > > requirement. Either into a type boolean (my preference), or putting no > > restrictions on the Parquet level, allowing all IEEE 754 float numbers. > If > > some top-level application wants to put additional constraints on the > data > > type, like Iceberg not allowing NaN/infinity, it can still do so one > layer > > above. It seems like this was the biggest point of discussion remaining. > > All these options work for me, and we should move forward with this > > proposal. > > > > Concerning additional element types like bf16, they can be introduced as > > Parquet types (at least in the current LIST-based spec proposal). > > > > Concerning Will’s suggestion around representing VECTOR as a variable > > length list with a MIN/MAX_LENGTH metadata restriction: I think that a > > dedicated logical VECTOR type still provides enough benefits by > describing > > the semantics of the data, and allowing to capture future changes around > > statistics. In that case, the length is better captured as an exact > length > > for this type specifically. I’m also not sure about other use cases for > > MIN_LENGTH, so introducing a new type constraint system just for the > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my > > view. > > > > Best, > > Philipp > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > > [email protected]> > > wrote: > > > > > +1 Sebastian, a vector type suggests usage beyond normalized embeddings > > and > > > engines really aren't going to be able to elide all the checks anyway. > > > > > > Antoine, you suggested we explore metadata solutions too :) > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this > in > > > Parquet. Engines can write opaque blobs they understand in the > metadata > > > (e.g. Arrow does) so the extra type info round-trips, but this isn't > > > standardized, portable, nor interoperable. > > > > > > So perhaps standardizing constraint metadata or new Thrift fields (my > > pref) > > > can generically support VECTOR(N) whilst also being more broadly useful > > for > > > other use cases and data types? > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH > > for > > > Parquet's variable-length physcial types. These would count logical > > units > > > (e.g. UTF8 code points for STRING) rather than bytes. > > > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - > > can > > > pick up these constraints if they support doing so. We don't actually > > need > > > a logical type to express a VECTOR then. > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed > on > > > UTF8 columns etc. > > > > > > This also seems generally useful in the future as we approach > expressing > > > other tensor data shapes and types. > > > > > > This won't change the mechanics: the storage type doesn't enforce these > > > constraints and engines must check them. It's previously established > that > > > performance is just a software problem and there is no fundamental > > > performance advantage to the fixed size types in Dremel-like formats > like > > > Parquet. > > > > > > Best, > > > Will > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > > [email protected] > > > > > > > wrote: > > > > > > > After listening in on the last meeting Wednesday, I have been > thinking > > > > about this quite a bit, and thought I would share my 2 cents. > > > > > > > > I suggest not requiring VECTOR elements to be finite, as proposed in > > the > > > PR > > > > <https://github.com/apache/parquet-format/pull/624>. > > > > > > > > While Divjot and others note the specialization for UTF-8, Parquet > > > already > > > > defines DOUBLE and FLOAT as IEEE encodings > > > > < > https://github.com/apache/parquet-format/blob/master/README.md#types > > >. > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction > > on > > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns > have > > > > never had such a rule. > > > > > > > > It would also not match how the systems that actually produce these > > > vectors > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, > NumPy, > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. > > > > > > > > pgvector's "all elements must be finite" rule is an insert policy. > > While > > > I > > > > think Parquet as a general-purpose format should allow them out of > the > > > box. > > > > It is a user system decision to disallow writing these values or to > > throw > > > > errors when encountering them. > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector math. > Finite > > > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, > > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. > > PyTorch > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in > > fp32. > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR > > > itself > > > > was required to be finite. The ban only forbids storing specials in > the > > > > coordinates, and does not prevent even simple downstream computations > > > from > > > > producing them. > > > > > > > > I suggest keeping VECTOR as the type and add an optional bool > > > > VectorType.finite (default false). Engines that want pgvector-like > > > > rejection can require the flag = true. Engines that use NumPy / > > PyTorch / > > > > Spark arrays can keep representable IEEE values. For the same > reason, I > > > do > > > > not favor adding yet another type called FINITE_VECTOR. > > > > > > > > Best, > > > > Sebastian > > > > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > > > [email protected]> > > > > wrote: > > > > > > > > > I agree with moving forward with the proposal and renaming VECTOR > to > > > > > FINITE_VECTOR or something > > > > > similar if others feel that better communicates its intent. I see > > value > > > > in > > > > > having a vector type that guarantees > > > > > its values are mathematically-correct vectors for a few reasons: > > > > > > > > > > 1. As others have mentioned, this is similar to the STRING logical > > type > > > > > mandating UTF-8. > > > > > 2. It is not possible to rely on statistics for detecting NaN or > > +-inf > > > > > values as all statistics are optional. > > > > > To my knowledge, there is no precedent for requiring that writers > > emit > > > > > stats for specific logical types. > > > > > 3. Without statistics, NaN/inf values could flow through the engine > > and > > > > > block them for specific functionality. > > > > > This seems to be Lance's behavior. However, I don't think this is > as > > > easy > > > > > as it sounds and Lance has run > > > > > into several bugs due to this ([1] [2] for examples). Their Python > > [3] > > > > and > > > > > Rust [4] APIs default to erroring on > > > > > vectors that contain NaN values. > > > > > > > > > > I agree that Parquet is a general-purpose format, but I don't think > > > this > > > > > implies the lack of domain-specific types. > > > > > Instead, this broadens the use cases where Parquet would be > > considered > > > a > > > > > good fit. I also fully agree with > > > > > continuing work on a general fixed-size list type if we believe > there > > > are > > > > > non-vector use cases that would benefit > > > > > from such guarantees at the storage layer. > > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 > > in > > > > > "root > > > > > cause") > > > > > [3] > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > [4] > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > I also favor moving forward with the proposal as is. I see strong > > > value > > > > > and > > > > > > use cases for the numeric vector type, and I agree with the > > > > restrictions > > > > > > around NaN/Inf values and element types in general. Precedent > > exists > > > in > > > > > > other types that restrict which physical type values are > logically > > > > valid > > > > > > (e.g., the geospatial types needing to be WKB-encoded). A > general > > > > > > fixed-size list also has value, but I think it makes sense to > > > separate > > > > > the > > > > > > two types. The semantics are relevant, and I can see how the two > > > types > > > > > > might diverge further in the future regarding statistics or > > > encodings. > > > > > > > > > > > > I would be okay with renaming the vector type, but it seems like > > the > > > > > VECTOR > > > > > > name is already used across other engines mentioned in this > thread. > > > > > > > > > > > > Best, > > > > > > Philipp > > > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> > wrote: > > > > > > > > > > > > > To separate orthogonal concerns, I feel we should treat the > > general > > > > > > > FixedSizeList as the underlying physical layout, and separate > it > > > from > > > > > the > > > > > > > discussion of a specialized VECTOR (or broader ML tensor) > logical > > > > type. > > > > > > > > > > > > > > Currently, the immediate need is to define a specialized VECTOR > > > type > > > > so > > > > > > > that different engines and formats have a common semantic > > contract > > > > for > > > > > > > interoperability. > > > > > > > > > > > > > > If the main point of contention is just the NaN/Inf/null > > > limitations, > > > > > > maybe > > > > > > > we can resolve this by adding a qualification to the name (like > > > > Andrew > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid > > > > confusion > > > > > > with > > > > > > > broader ML use cases. Alternatively, we could introduce a > boolean > > > > flag > > > > > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > I am a little confused about the points of disagreement here. > > > > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the specific > > use > > > > case > > > > > > of > > > > > > > > fixed length arrays with non-finite values used for vector > > > > > embeddings? > > > > > > > If > > > > > > > > so, maybe we could add a qualification to the name > > > (FINITE_VECTOR? > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > > > > > It seems like there is precedent for defining logical types > > that > > > > > > specify > > > > > > > > certain physical types and restrict their contents to help > > > readers > > > > > > > > interpret meaning, for example string[1] or JSON; Defining > one > > > for > > > > > > > vector > > > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits their > usage, > > > but > > > > > > we're > > > > > > > > > > talking about a general-purpose file format that aims to > be > > > > > broadly > > > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general purpose, > > but > > > > > types > > > > > > > > within > > > > > > > > > the format define specific representation and semantics to > > > > achieve > > > > > > that > > > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > > > trivial > > > > > to > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy > or > > > > Pandas > > > > > > (or > > > > > > > > > > any other piece of dataframe or scientific computing > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite values is > > > directly > > > > > > > related > > > > > > > > to > > > > > > > > > the semantic meaning of a vector. Vectors have specific > > > > > > > characteristics > > > > > > > > > (dimension, magnitude, direction, etc). Non-finite values > > are > > > > > > invalid > > > > > > > > and > > > > > > > > > break those characteristics. I'd liken it to storing > > non-utf8 > > > > > values > > > > > > > in > > > > > > > > a > > > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > > > > > That also differentiates a vector from a fixed length list, > > > which > > > > > is > > > > > > > more > > > > > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > > > > > > > > > I also don't think we need to bundle all similarly shaped > use > > > > cases > > > > > > > into > > > > > > > > > the same solution (e.g. flxed size list). Vector is a well > > > > > > established > > > > > > > > > data type at this point and aligns with the general purpose > > > > nature > > > > > of > > > > > > > the > > > > > > > > > format (many other formats and databases natively support > > > this). > > > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > > > > I think something I haven't seen yet is the clear > > use-case > > > > for > > > > > > > > > non-finite > > > > > > > > > > > values within a vector type. There has been a lot of > > > > > theoretical > > > > > > > > > > discussion > > > > > > > > > > > here about use cases but what toolings currently > produce > > > > these > > > > > > > > vectors > > > > > > > > > > and > > > > > > > > > > > how do they function in downstream engines? I'd like to > > > limit > > > > > > > > > introducing > > > > > > > > > > > types that have a wide range of values which are simply > > are > > > > > > failure > > > > > > > > > modes > > > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > > > trivial > > > > > to > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy > or > > > > Pandas > > > > > > (or > > > > > > > > > > any other piece of dataframe or scientific computing > > > software). > > > > > > > > > > > > > > > > > > > > It's not like the general concept of a vector is a IETF > > > > standard > > > > > or > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the > values > > > are > > > > > > > expected > > > > > > > > > > to be valid according to the respective specs) do not > > apply. > > > > > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance of > > converging > > > > if > > > > > we > > > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet > > > (with > > > > a > > > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > > > > > 2. representing the information that a given column (or > > > column > > > > > > chunk, > > > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > > > > > ... both of which are useful on their own right, but do > not > > > > have > > > > > to > > > > > > > be > > > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
