A bit late to the party, but here we go: I have the strong sentiment that the FINITE_VECTOR type as it is currently being discussed will not be very useful and just increase implementation complexity. The arguments for the type are all quite weak (details see below). The FINITE part doesn't seem to yield any value and is not a useful restriction by itself. The vector (i.e. fixed length) part should rather be generalized instead of making a one-off thing. In general, the existing types are already working well for uncompressed vectors. The real challenge that would bring real benefits is supporting quantization. This is where you get orders of magnitude improvements to storage space and processing speed.
Let's first discuss the FINITE part. So far, the only arguments I have read for this are: 1. many other vector databases have it 2. vectors are the hot stuff right now, so Parquet better boast having "native" type support for vectors 3. by focussing on this common special case, we can make rapid progress, while a general solution would take too long to specify 4. to give a guarantee to readers that no INF/NaN/NULL will be in the vector the first two arguments are quite weak by themselves. Vector databases are very specialized engines. Such engines can of course be very opinionated on their data types. A general format like Parquet must instead weigh whether a special type is worth the added complexity over a more general, widely applicable type. In this case, it is IMHO not at all. And just having a type called "vector" for advertisement reasons to make Parquet appear more modern seems a non-goal in the first place. The third "time to market" argument is valid for systems that don't need a stable interface. It is a very dangerous attitude for formats like Parquet that must be stable long term and should have a consolidated, concise, non-redundant type system. For such long term storage formats, we need to take the time to build the right thing, as any shortcuts we take will stay as blemishes in the format forever. If I read the discussion correctly, this is actually the main argument as of now: We have found that it is hard to align on how a fixed size list type should look like, so we want to go the easier route of a specialized type. This seems like a dangerous shortcut fueled by feature recency bias. Ask yourself: Would we want this, if Parquet already had a fixed size list type? So finally, the only technical argument is the fourth one: "It signals semantics and ensures that no bad data lands in the format". I see two big problems with this: 1. Readers *still* need to test for NaN / INF because Parquet files may be corrupt. Nothing can stop a malicious or buggy writer from physically writing NaN or INF. *So you save no single line of code in a reader. You save no branches, as you still need to handle the INF/NaN case.* So this doesn't help with performance at all. The only difference is, that the error message changes from "found an invalid INF value in the data" to "Your parquet file seems to be corrupt". It is actually quite the opposite, readers and writers get more complex having to handle this new type. If you now argue that it's not about performance, it's about defined semantics and erroring out early in writers, then the second problem arises: 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple guarantees that you need for sensible vector processing. If "no INF/NaN/NULL" was the only guarantee you needed, I would agree that FINITE_VECTOR is useful, but it is by no means the only guarantee you need, instead, depending on the distance metric, you also need the following guarantees. E.g.: - No vector may be the 0 vector, otherwise, e.g., cosine similarity is undefined. Cosine similarity is often used as default, so this is actually quite important in practice. - If we want to use a dot product, then all vectors need to guarantee a length of 1. This would actually be quite useful for performance reasons. - The vectors must come from the same model as the vector used for the search. Otherwise, the search yields a numeric result, but that result has no sensible meaning. So, what use is a type that just guarantees part of the things you need for vector search but not others? You still need additional guarantees in addition to that type. What makes INF/NaN so important in comparison to, say, "no 0 vector present", that it warrants an own type, while the other guarantees do not? If we really want this type as a basis for cross-engine sharing of vectors, then it needs to give *all* guarantees. I'm not against a specialized vector type if it really carries *all* information to be useful by itself. If it carried all that information, then a reading engine could indeed validate that only meaningful vector searches are executed. But as long as it is just a half-baked thing, it is not useful by itself and just a burden on each implementer and maintainer. If we need extra information anyway, why not just stick to a general type like VECTOR (without FINITE). If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what advantage you will *really* get over just a (potentially fixed size) List of Float? As discussed, it's not performance. It's also not a semantic marker that is sufficient by itself. It's just a small hint that allows writers to error out once they see a NaN or INF in the input, while they will still happily write a bogus 0 vector. This seems to be of zero benefit. Anecdotally, we currently handle petabytes of vectors in production in Data360. We simply store them in a List of Float, with an optimized R-Level reader implementation that rapidly checks that all vectors are the same length. This solution works just fine. The overhead of R-level decoding is barely measurable. The check is super fast and R-levels are tiny in comparison to the payload vectors, so also the added I/O is negligible. This is also why I'm so firm in my sentiment: We are running a production grade system processing trillions of vectors right now and see no big issues with the types that Parquet currently provides. As I don't just want to be a naysayer and point out problems with the current design, here are the directions that I would propose indeed: - a general fixed size list type. Don't do shortcuts just because the discussion is too tedious. - a vector type that indeed carries enough information to be really useful for vector search by itself. (e.g., model name / version, optional guarantees for no 0 vector / normalized vector length). With such a type, you could really do well defined vector searches even if the writing engine is completely different to the reading one. - Standardize quantizations. This will be a real game changer for performance. Cheers, Jan Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <[email protected]>: > I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a good > idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural. > Should we move forward with proposing FINITE_VECTOR/VECTOR first? > > On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < > [email protected]> wrote: > > > I suggest we go with VECTOR for the finite vector to keep the name the > same > > as other engines/type systems. If we want a non-finite one in the future > we > > can name that something else or we add fixed size list that will serve > the > > same purpose without a special name. > > > > > > On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> wrote: > > > > > A logical type would: > > > - Describe data for users and readers to understand what the file > > contains > > > - Signal that repetition and definition levels will reflect a fixed > > number > > > of elements, which implementations can use to optimize decoding > > > - Guarantee that there are no null elements (because the type cannot be > > > applied to a list with optional elements) > > > - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values > > so > > > that the vectors have a finite length and are suitable for distance > > > calculations > > > > > > The finite value restriction is a restriction on writers to prevent bad > > > data from being written. > > > > > > Ignoring logical types that are not understood is a rule for readers. A > > > reader can handle vectors and pass it on to another layer. For > instance, > > a > > > Parquet reader that doesn't support geometry passes values as `binary`. > > > That binary value can be passed to a geometry UDF, even if the > underlying > > > engine and Parquet reader doesn't have support. > > > > > > A tool that doesn't understand FINITE_VECTOR can similarly pass a list > of > > > floats along to a UDF. But it is still important to the user that bad > > > values are not present in the data. In your example, bad values are > > vectors > > > that cause the centroid calculation to fail, like vectors with infinite > > > length. > > > > > > Ryan > > > > > > On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < > > > [email protected]> > > > wrote: > > > > > > > Could we clarify the use cases for a VECTOR annotation on > LIST<FLOAT>? > > > > Specifically, what does it make possible that's impossible today, and > > > what > > > > does it make easier that's currently hard? > > > > > > > > I'm struggling to find a strong answer. I'd like every engine to > offer > > > > vector functions on lists of floats without needing a new logical > type. > > > > > > > > To be clear, I'm still enthusiastic about a vector type that > describes > > > how > > > > to unpack and scale composite representations (quantised codes plus > > > scales, > > > > codebooks, etc.), in the direction ML is going. That adds information > > > > engines don't have. A label on LIST<FLOAT> doesn't: engines already > > > > understand the physical type, and many already offer vector functions > > on > > > > it. I don't see a performance benefit, and NaN/null/Inf checks are > > cheap > > > > enough that engines should do them anyway. The label also doesn't say > > > what > > > > you can do with the column. Knowing that a column holds embeddings > > > doesn't > > > > say how they can be used, just as zipcode and age are both integers > but > > > > multiplying them is meaningless. > > > > > > > > My concern is the opposite effect. We're establishing that engines > > should > > > > ignore logical types they don't understand. So a tool that doesn't > know > > > > VECTOR will read the column as LIST<FLOAT>, let the user compute > > > centroids > > > > (say), and write plain LIST<FLOAT> back out. If other engines > restrict > > > > embedding functions to VECTOR-annotated columns, that file stops > > working > > > > with them, through no fault of the user. An annotation that doesn't > > > change > > > > what the data is shouldn't be able to make it unusable. > > > > > > > > Am I missing a use case? > > > > > > > > Best, Will > > > > > > > > On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> wrote: > > > > > > > > > I also think a FINITE_VECTOR type brings a lot of values to many > use > > > > cases. > > > > > It is would be great to make progress on the FINITE_VECTOR, and not > > to > > > > fit > > > > > every use cases into the FIXED_LENGTH_LIST. > > > > > > > > > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> wrote: > > > > > > > > > > > Going through this thread again, it looks like there are a few > > things > > > > > that > > > > > > people generally agree on: > > > > > > > > > > > > - In the short term, we want to introduce a logical type or > types; > > a > > > > > better > > > > > > physical representation can be done later. We can focus on > logical > > > > types. > > > > > > - There is precedent in logical types for restricting values, > like > > > UTF8 > > > > > > strings to contain only UTF-8 codepoints. I think we also agree > > that > > > we > > > > > > could restrict vector elements to non-nested or primitive types. > > > > > > - There are many examples of databases that restrict their vector > > > types > > > > > to > > > > > > non-null, finite elements. > > > > > > - Some people want to have write-time guarantees that vectors > have > > > only > > > > > > non-null, finite elements > > > > > > - Some people want to extend this work to include tensors, > shorter > > > > > floating > > > > > > point representations, variable lengths, NaN or Infinity > elements, > > > etc. > > > > > > > > > > > > It sounds like we are nearly there, but have two halves of the > > > > community > > > > > > that care about different things. That just means we should have > > > > multiple > > > > > > types. It's entirely reasonable to want a type that allows NaN > > > values, > > > > > and > > > > > > it's also reasonable for those of us building databases, table > > > > standards, > > > > > > and vector indexes to care about guarantees that elements are > > finite. > > > > > > There's more than enough examples to show that this is a > legitimate > > > use > > > > > > case. > > > > > > > > > > > > Since I don't see the concerns of the different use cases > changing, > > > it > > > > > > looks like we need to find a compromise to move forward. I > propose > > > that > > > > > we > > > > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST > > or > > > > > > another VECTOR type that allows infinite and NaN values. > > > > > > > > > > > > Can we agree on a compromise? Is there a strong reason to try to > > fit > > > > > these > > > > > > use cases into a single type? What if that means not making > > progress? > > > > > > > > > > > > Ryan > > > > > > > > > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Thinking about this more, I am also fine with pulling out the > > > > “NaN/Inf” > > > > > > > requirement. Either into a type boolean (my preference), or > > putting > > > > no > > > > > > > restrictions on the Parquet level, allowing all IEEE 754 float > > > > numbers. > > > > > > If > > > > > > > some top-level application wants to put additional constraints > on > > > the > > > > > > data > > > > > > > type, like Iceberg not allowing NaN/infinity, it can still do > so > > > one > > > > > > layer > > > > > > > above. It seems like this was the biggest point of discussion > > > > > remaining. > > > > > > > All these options work for me, and we should move forward with > > this > > > > > > > proposal. > > > > > > > > > > > > > > Concerning additional element types like bf16, they can be > > > introduced > > > > > as > > > > > > > Parquet types (at least in the current LIST-based spec > proposal). > > > > > > > > > > > > > > Concerning Will’s suggestion around representing VECTOR as a > > > variable > > > > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think > > > that > > > > a > > > > > > > dedicated logical VECTOR type still provides enough benefits by > > > > > > describing > > > > > > > the semantics of the data, and allowing to capture future > changes > > > > > around > > > > > > > statistics. In that case, the length is better captured as an > > exact > > > > > > length > > > > > > > for this type specifically. I’m also not sure about other use > > cases > > > > for > > > > > > > MIN_LENGTH, so introducing a new type constraint system just > for > > > the > > > > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough > value > > in > > > > my > > > > > > > view. > > > > > > > > > > > > > > Best, > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > +1 Sebastian, a vector type suggests usage beyond normalized > > > > > embeddings > > > > > > > and > > > > > > > > engines really aren't going to be able to elide all the > checks > > > > > anyway. > > > > > > > > > > > > > > > > Antoine, you suggested we explore metadata solutions too :) > > > > > > > > > > > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't > > express > > > > this > > > > > > in > > > > > > > > Parquet. Engines can write opaque blobs they understand in > the > > > > > > metadata > > > > > > > > (e.g. Arrow does) so the extra type info round-trips, but > this > > > > isn't > > > > > > > > standardized, portable, nor interoperable. > > > > > > > > > > > > > > > > So perhaps standardizing constraint metadata or new Thrift > > fields > > > > (my > > > > > > > pref) > > > > > > > > can generically support VECTOR(N) whilst also being more > > broadly > > > > > useful > > > > > > > for > > > > > > > > other use cases and data types? > > > > > > > > > > > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and > > > > > MAX_LENGTH > > > > > > > for > > > > > > > > Parquet's variable-length physcial types. These would count > > > > logical > > > > > > > units > > > > > > > > (e.g. UTF8 code points for STRING) rather than bytes. > > > > > > > > > > > > > > > > Then one engine's VECTOR(N) can be written to Parquet as > > LIST(N) > > > > with > > > > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different > > > > > engines - > > > > > > > can > > > > > > > > pick up these constraints if they support doing so. We don't > > > > > actually > > > > > > > need > > > > > > > > a logical type to express a VECTOR then. > > > > > > > > > > > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be > > > > > expressed > > > > > > on > > > > > > > > UTF8 columns etc. > > > > > > > > > > > > > > > > This also seems generally useful in the future as we approach > > > > > > expressing > > > > > > > > other tensor data shapes and types. > > > > > > > > > > > > > > > > This won't change the mechanics: the storage type doesn't > > enforce > > > > > these > > > > > > > > constraints and engines must check them. It's previously > > > > established > > > > > > that > > > > > > > > performance is just a software problem and there is no > > > fundamental > > > > > > > > performance advantage to the fixed size types in Dremel-like > > > > formats > > > > > > like > > > > > > > > Parquet. > > > > > > > > > > > > > > > > Best, > > > > > > > > Will > > > > > > > > > > > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > After listening in on the last meeting Wednesday, I have > been > > > > > > thinking > > > > > > > > > about this quite a bit, and thought I would share my 2 > cents. > > > > > > > > > > > > > > > > > > I suggest not requiring VECTOR elements to be finite, as > > > proposed > > > > > in > > > > > > > the > > > > > > > > PR > > > > > > > > > <https://github.com/apache/parquet-format/pull/624>. > > > > > > > > > > > > > > > > > > While Divjot and others note the specialization for UTF-8, > > > > Parquet > > > > > > > > already > > > > > > > > > defines DOUBLE and FLOAT as IEEE encodings > > > > > > > > > < > > > > > > > > https://github.com/apache/parquet-format/blob/master/README.md#types > > > > > > > >. > > > > > > > > > > > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new > > > > > restriction > > > > > > > on > > > > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE > > > columns > > > > > > have > > > > > > > > > never had such a rule. > > > > > > > > > > > > > > > > > > It would also not match how the systems that actually > produce > > > > these > > > > > > > > vectors > > > > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, > > DuckDB, > > > > > > NumPy, > > > > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and > > OpenBLAS. > > > > > > > > > > > > > > > > > > pgvector's "all elements must be finite" rule is an insert > > > > policy. > > > > > > > While > > > > > > > > I > > > > > > > > > think Parquet as a general-purpose format should allow them > > out > > > > of > > > > > > the > > > > > > > > box. > > > > > > > > > It is a user system decision to disallow writing these > values > > > or > > > > to > > > > > > > throw > > > > > > > > > errors when encountering them. > > > > > > > > > > > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector > > math. > > > > > > Finite > > > > > > > > > coordinates still overflow. For instance: magnitude = > > > > ||[FLOAT_MAX, > > > > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to > > > +Inf. > > > > > > > PyTorch > > > > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is > > inf > > > > in > > > > > > > fp32. > > > > > > > > > > > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while > the > > > > > VECTOR > > > > > > > > itself > > > > > > > > > was required to be finite. The ban only forbids storing > > > specials > > > > in > > > > > > the > > > > > > > > > coordinates, and does not prevent even simple downstream > > > > > computations > > > > > > > > from > > > > > > > > > producing them. > > > > > > > > > > > > > > > > > > I suggest keeping VECTOR as the type and add an optional > bool > > > > > > > > > VectorType.finite (default false). Engines that want > > > > pgvector-like > > > > > > > > > rejection can require the flag = true. Engines that use > > NumPy / > > > > > > > PyTorch / > > > > > > > > > Spark arrays can keep representable IEEE values. For the > same > > > > > > reason, I > > > > > > > > do > > > > > > > > > not favor adding yet another type called FINITE_VECTOR. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Sebastian > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I agree with moving forward with the proposal and > renaming > > > > VECTOR > > > > > > to > > > > > > > > > > FINITE_VECTOR or something > > > > > > > > > > similar if others feel that better communicates its > > intent. I > > > > see > > > > > > > value > > > > > > > > > in > > > > > > > > > > having a vector type that guarantees > > > > > > > > > > its values are mathematically-correct vectors for a few > > > > reasons: > > > > > > > > > > > > > > > > > > > > 1. As others have mentioned, this is similar to the > STRING > > > > > logical > > > > > > > type > > > > > > > > > > mandating UTF-8. > > > > > > > > > > 2. It is not possible to rely on statistics for detecting > > NaN > > > > or > > > > > > > +-inf > > > > > > > > > > values as all statistics are optional. > > > > > > > > > > To my knowledge, there is no precedent for requiring that > > > > writers > > > > > > > emit > > > > > > > > > > stats for specific logical types. > > > > > > > > > > 3. Without statistics, NaN/inf values could flow through > > the > > > > > engine > > > > > > > and > > > > > > > > > > block them for specific functionality. > > > > > > > > > > This seems to be Lance's behavior. However, I don't think > > > this > > > > is > > > > > > as > > > > > > > > easy > > > > > > > > > > as it sounds and Lance has run > > > > > > > > > > into several bugs due to this ([1] [2] for examples). > Their > > > > > Python > > > > > > > [3] > > > > > > > > > and > > > > > > > > > > Rust [4] APIs default to erroring on > > > > > > > > > > vectors that contain NaN values. > > > > > > > > > > > > > > > > > > > > I agree that Parquet is a general-purpose format, but I > > don't > > > > > think > > > > > > > > this > > > > > > > > > > implies the lack of domain-specific types. > > > > > > > > > > Instead, this broadens the use cases where Parquet would > be > > > > > > > considered > > > > > > > > a > > > > > > > > > > good fit. I also fully agree with > > > > > > > > > > continuing work on a general fixed-size list type if we > > > believe > > > > > > there > > > > > > > > are > > > > > > > > > > non-vector use cases that would benefit > > > > > > > > > > from such guarantees at the storage layer. > > > > > > > > > > > > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > > > > > > > > [2] https://github.com/lance-format/lance/issues/8607 > (see > > > > > point 2 > > > > > > > in > > > > > > > > > > "root > > > > > > > > > > cause") > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > > > > > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via > dev < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > I also favor moving forward with the proposal as is. I > > see > > > > > strong > > > > > > > > value > > > > > > > > > > and > > > > > > > > > > > use cases for the numeric vector type, and I agree with > > the > > > > > > > > > restrictions > > > > > > > > > > > around NaN/Inf values and element types in general. > > > Precedent > > > > > > > exists > > > > > > > > in > > > > > > > > > > > other types that restrict which physical type values > are > > > > > > logically > > > > > > > > > valid > > > > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded). > > A > > > > > > general > > > > > > > > > > > fixed-size list also has value, but I think it makes > > sense > > > to > > > > > > > > separate > > > > > > > > > > the > > > > > > > > > > > two types. The semantics are relevant, and I can see > how > > > the > > > > > two > > > > > > > > types > > > > > > > > > > > might diverge further in the future regarding > statistics > > or > > > > > > > > encodings. > > > > > > > > > > > > > > > > > > > > > > I would be okay with renaming the vector type, but it > > seems > > > > > like > > > > > > > the > > > > > > > > > > VECTOR > > > > > > > > > > > name is already used across other engines mentioned in > > this > > > > > > thread. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > To separate orthogonal concerns, I feel we should > treat > > > the > > > > > > > general > > > > > > > > > > > > FixedSizeList as the underlying physical layout, and > > > > separate > > > > > > it > > > > > > > > from > > > > > > > > > > the > > > > > > > > > > > > discussion of a specialized VECTOR (or broader ML > > tensor) > > > > > > logical > > > > > > > > > type. > > > > > > > > > > > > > > > > > > > > > > > > Currently, the immediate need is to define a > > specialized > > > > > VECTOR > > > > > > > > type > > > > > > > > > so > > > > > > > > > > > > that different engines and formats have a common > > semantic > > > > > > > contract > > > > > > > > > for > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > > > If the main point of contention is just the > > NaN/Inf/null > > > > > > > > limitations, > > > > > > > > > > > maybe > > > > > > > > > > > > we can resolve this by adding a qualification to the > > name > > > > > (like > > > > > > > > > Andrew > > > > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) > to > > > > avoid > > > > > > > > > confusion > > > > > > > > > > > with > > > > > > > > > > > > broader ML use cases. Alternatively, we could > > introduce a > > > > > > boolean > > > > > > > > > flag > > > > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed > > > > earlier. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I am a little confused about the points of > > disagreement > > > > > here. > > > > > > > > > > > > > > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the > > > > > specific > > > > > > > use > > > > > > > > > case > > > > > > > > > > > of > > > > > > > > > > > > > fixed length arrays with non-finite values used for > > > > vector > > > > > > > > > > embeddings? > > > > > > > > > > > > If > > > > > > > > > > > > > so, maybe we could add a qualification to the name > > > > > > > > (FINITE_VECTOR? > > > > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > > > > > > > > > > > > > > > It seems like there is precedent for defining > logical > > > > types > > > > > > > that > > > > > > > > > > > specify > > > > > > > > > > > > > certain physical types and restrict their contents > to > > > > help > > > > > > > > readers > > > > > > > > > > > > > interpret meaning, for example string[1] or JSON; > > > > Defining > > > > > > one > > > > > > > > for > > > > > > > > > > > > vector > > > > > > > > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits > > their > > > > > > usage, > > > > > > > > but > > > > > > > > > > > we're > > > > > > > > > > > > > > > talking about a general-purpose file format > that > > > aims > > > > > to > > > > > > be > > > > > > > > > > broadly > > > > > > > > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general > > > > > purpose, > > > > > > > but > > > > > > > > > > types > > > > > > > > > > > > > within > > > > > > > > > > > > > > the format define specific representation and > > > semantics > > > > > to > > > > > > > > > achieve > > > > > > > > > > > that > > > > > > > > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and > > it's > > > > > quite > > > > > > > > > trivial > > > > > > > > > > to > > > > > > > > > > > > > > > produce vectors with non-finite values using > e.g. > > > > NumPy > > > > > > or > > > > > > > > > Pandas > > > > > > > > > > > (or > > > > > > > > > > > > > > > any other piece of dataframe or scientific > > > computing > > > > > > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite > > values > > > is > > > > > > > > directly > > > > > > > > > > > > related > > > > > > > > > > > > > to > > > > > > > > > > > > > > the semantic meaning of a vector. Vectors have > > > > specific > > > > > > > > > > > > characteristics > > > > > > > > > > > > > > (dimension, magnitude, direction, etc). > Non-finite > > > > > values > > > > > > > are > > > > > > > > > > > invalid > > > > > > > > > > > > > and > > > > > > > > > > > > > > break those characteristics. I'd liken it to > > storing > > > > > > > non-utf8 > > > > > > > > > > values > > > > > > > > > > > > in > > > > > > > > > > > > > a > > > > > > > > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > > > > > > > > > > > > > > > That also differentiates a vector from a fixed > > length > > > > > list, > > > > > > > > which > > > > > > > > > > is > > > > > > > > > > > > more > > > > > > > > > > > > > > general and does not carry the same semantic > > meaning. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I also don't think we need to bundle all > similarly > > > > shaped > > > > > > use > > > > > > > > > cases > > > > > > > > > > > > into > > > > > > > > > > > > > > the same solution (e.g. flxed size list). Vector > > is > > > a > > > > > well > > > > > > > > > > > established > > > > > > > > > > > > > > data type at this point and aligns with the > general > > > > > purpose > > > > > > > > > nature > > > > > > > > > > of > > > > > > > > > > > > the > > > > > > > > > > > > > > format (many other formats and databases natively > > > > support > > > > > > > > this). > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit > : > > > > > > > > > > > > > > > > I think something I haven't seen yet is the > > clear > > > > > > > use-case > > > > > > > > > for > > > > > > > > > > > > > > non-finite > > > > > > > > > > > > > > > > values within a vector type. There has been a > > lot > > > > of > > > > > > > > > > theoretical > > > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > here about use cases but what toolings > > currently > > > > > > produce > > > > > > > > > these > > > > > > > > > > > > > vectors > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > how do they function in downstream engines? > I'd > > > > like > > > > > to > > > > > > > > limit > > > > > > > > > > > > > > introducing > > > > > > > > > > > > > > > > types that have a wide range of values which > > are > > > > > simply > > > > > > > are > > > > > > > > > > > failure > > > > > > > > > > > > > > modes > > > > > > > > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and > > it's > > > > > quite > > > > > > > > > trivial > > > > > > > > > > to > > > > > > > > > > > > > > > produce vectors with non-finite values using > e.g. > > > > NumPy > > > > > > or > > > > > > > > > Pandas > > > > > > > > > > > (or > > > > > > > > > > > > > > > any other piece of dataframe or scientific > > > computing > > > > > > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It's not like the general concept of a vector > is > > a > > > > IETF > > > > > > > > > standard > > > > > > > > > > or > > > > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where > > the > > > > > > values > > > > > > > > are > > > > > > > > > > > > expected > > > > > > > > > > > > > > > to be valid according to the respective specs) > do > > > not > > > > > > > apply. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance > > of > > > > > > > converging > > > > > > > > > if > > > > > > > > > > we > > > > > > > > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data > in > > > > > Parquet > > > > > > > > (with > > > > > > > > > a > > > > > > > > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. representing the information that a given > > column > > > > (or > > > > > > > > column > > > > > > > > > > > chunk, > > > > > > > > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ... both of which are useful on their own > right, > > > but > > > > do > > > > > > not > > > > > > > > > have > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
