I feel this discussion is getting overly academic around domain-specific requirements. As a general-purpose file format, Parquet should focus on defining clear, foundational data properties rather than trying to capture or enforce every application-level invariant in the type system.
If we look at existing logical types in Parquet, there is a clear precedent for how boundaries are drawn: 1) STRING requires valid UTF-8, GEOMETRY and GEOGRAPHY require valid WKB, and JSON and BSON require valid documents. 2) In all of these cases, the underlying physical storage cannot physically prevent a buggy writer from emitting invalid bytes. The logical type establishes a clear baseline contract, while validation remains the responsibility of writer and reader implementations. 3) Higher-level domain semantics (such as whether a vector is non-zero for cosine similarity, normalized to unit length for dot product, or tied to a specific embedding model) belong to specific applications, rather than the Parquet logical type definition. Defining a vector logical type with a fixed length and finite, non-null primitive elements sets a pragmatic, well-understood boundary that aligns with mainstream engines without over-engineering the format. On Thu, Oct 15, 2026 at 10:28 AM <[email protected]> wrote: > Hi Everyone, > > Glad to see a Vector type being added. I have a few use cases I want to > pursue once we have an agreed upon definition. > > In regards to that definition discussion. I feel comfortable with the idea > of the semantic meaning Dan raises though I can’t shake a slight discomfort > (maybe FOMO) with limiting values in the physical definition. > > A few naive questions: > 1. Would a physical definition with NANs & Inf+/- allow efficient > encodings like ALP? > 2. If the physical representation allows values that make encodings like > ALP impossible, can a tighter Logical type allow different encodings? > 3. Can we RLE or binpack a vector? > > Best Regards, > Kurtis C. Wright > > On Oct 14, 2026, at 12:45, Daniel Weeks <[email protected]> wrote: > > > > Hey Jan, > > > > There's a lot that I disagree with respect to the framing of these > > arguments and how we should be thinking about introducing vector as a > > logical type. There's a repeated tendency to let the physical > > representation or performance drive discussion around types as opposed to > > what it represents. > > > > In parquet, establishing semantic meaning is the purpose of logical > types. > > The logical type separates how to interpret a value from its physical > > representation. We can represent a timestamp with INT64 or even a string > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and > VECTOR > > may have the exact same representation and performance, they have > different > > semantic meaning. Databases and engines across the ecosystem already > > recognize this differentiation and have separate representations. The > > choice to separate these isn't because the discussion is too tedious; > > they're fundamentally different things. > > > > The finite requirement is what makes a vector a reflection of what it > > represents. As we've discussed in the thread, non-finite values in the > > representation violate the properties of a vector and the value ceases to > > be a vector. On the other hand, a 0 vector is still a vector (you can > > argue the utility of it, but it is a valid vector). Implementations can > > choose what they validate or how to handle execution errors; that's an > > implementation concern, not a requirement. Operations performed on a > > vector depend on the context in which they are used, but the > representation > > doesn't need to solve scenarios where that application doesn't make > sense. > > > > The guarantees or context needed to use a vector do not need to be > inherent > > in the type for it to be a useful type. We don't need the currency used > > for decimal or whether a floating point is a normalized value as part of > > the type to make use of those types. This conflates the use cases with > the > > type system and unnecessarily attempts to draw the context into the type > > definition. Similarly, you don't need to capture normalization, model, > > etc. as part of the type for it to be useful. > > > > So the reason we want to add a vector type (or FINITE_VECTOR) is because > we > > want to store vectors with the semantic meaning and requirements based on > > what they represent. > > > > -Dan > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <[email protected]> wrote: > >> > >> A bit late to the party, but here we go: > >> > >> I have the strong sentiment that the FINITE_VECTOR type as it is > currently > >> being discussed will not be very useful and just increase implementation > >> complexity. The arguments for the type are all quite weak (details see > >> below). The FINITE part doesn't seem to yield any value and is not a > useful > >> restriction by itself. The vector (i.e. fixed length) part should > rather be > >> generalized instead of making a one-off thing. In general, the existing > >> types are already working well for uncompressed vectors. The real > challenge > >> that would bring real benefits is supporting quantization. This is where > >> you get orders of magnitude improvements to storage space and processing > >> speed. > >> > >> Let's first discuss the FINITE part. So far, the only arguments I have > read > >> for this are: > >> > >> 1. many other vector databases have it > >> 2. vectors are the hot stuff right now, so Parquet better boast having > >> "native" type support for vectors > >> 3. by focussing on this common special case, we can make rapid progress, > >> while a general solution would take too long to specify > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the > >> vector > >> > >> the first two arguments are quite weak by themselves. Vector databases > are > >> very specialized engines. Such engines can of course be very > opinionated on > >> their data types. A general format like Parquet must instead weigh > whether > >> a special type is worth the added complexity over a more general, widely > >> applicable type. In this case, it is IMHO not at all. And just having a > >> type called "vector" for advertisement reasons to make Parquet appear > more > >> modern seems a non-goal in the first place. > >> > >> The third "time to market" argument is valid for systems that don't > need a > >> stable interface. It is a very dangerous attitude for formats like > Parquet > >> that must be stable long term and should have a consolidated, concise, > >> non-redundant type system. For such long term storage formats, we need > to > >> take the time to build the right thing, as any shortcuts we take will > stay > >> as blemishes in the format forever. If I read the discussion correctly, > >> this is actually the main argument as of now: We have found that it is > hard > >> to align on how a fixed size list type should look like, so we want to > go > >> the easier route of a specialized type. This seems like a > >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would > we > >> want this, if Parquet already had a fixed size list type? > >> > >> So finally, the only technical argument is the fourth one: "It signals > >> semantics and ensures that no bad data lands in the format". I see two > big > >> problems with this: > >> > >> 1. Readers *still* need to test for NaN / INF because Parquet files may > be > >> corrupt. Nothing can stop a malicious or buggy writer from physically > >> writing NaN or INF. *So you save no single line of code in a reader. You > >> save no branches, as you still need to handle the INF/NaN case.* So this > >> doesn't help with performance at all. The only difference is, that the > >> error message changes from "found an invalid INF value in the data" to > >> "Your parquet file seems to be corrupt". It is actually quite the > opposite, > >> readers and writers get more complex having to handle this new type. > >> > >> If you now argue that it's not about performance, it's about defined > >> semantics and erroring out early in writers, then the second problem > >> arises: > >> > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple > >> guarantees that you need for sensible vector processing. If "no > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that > >> FINITE_VECTOR is useful, but it is by no means the only guarantee you > need, > >> instead, depending on the distance metric, you also need the following > >> guarantees. E.g.: > >> > >> - No vector may be the 0 vector, otherwise, e.g., cosine similarity is > >> undefined. Cosine similarity is often used as default, so this is > >> actually > >> quite important in practice. > >> - If we want to use a dot product, then all vectors need to guarantee > a > >> length of 1. This would actually be quite useful for performance > >> reasons. > >> - The vectors must come from the same model as the vector used for the > >> search. Otherwise, the search yields a numeric result, but that result > >> has > >> no sensible meaning. > >> > >> So, what use is a type that just guarantees part of the things you need > for > >> vector search but not others? You still need additional guarantees in > >> addition to that type. What makes INF/NaN so important in comparison to, > >> say, "no 0 vector present", that it warrants an own type, while the > other > >> guarantees do not? If we really want this type as a basis for > cross-engine > >> sharing of vectors, then it needs to give *all* guarantees. > >> > >> I'm not against a specialized vector type if it really carries *all* > >> information to be useful by itself. If it carried all that information, > >> then a reading engine could indeed validate that only meaningful vector > >> searches are executed. But as long as it is just a half-baked thing, it > is > >> not useful by itself and just a burden on each implementer and > maintainer. > >> If we need extra information anyway, why not just stick to a general > type > >> like VECTOR (without FINITE). > >> > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what > >> advantage you will *really* get over just a (potentially fixed size) > List > >> of Float? As discussed, it's not performance. It's also not a semantic > >> marker that is sufficient by itself. It's just a small hint that allows > >> writers to error out once they see a NaN or INF in the input, while they > >> will still happily write a bogus 0 vector. This seems to be of zero > >> benefit. > >> > >> Anecdotally, we currently handle petabytes of vectors in production in > >> Data360. We simply store them in a List of Float, with an optimized > R-Level > >> reader implementation that rapidly checks that all vectors are the same > >> length. This solution works just fine. The overhead of R-level decoding > is > >> barely measurable. The check is super fast and R-levels are tiny in > >> comparison to the payload vectors, so also the added I/O is negligible. > >> This is also why I'm so firm in my sentiment: We are running a > production > >> grade system processing trillions of vectors right now and see no big > >> issues with the types that Parquet currently provides. > >> > >> As I don't just want to be a naysayer and point out problems with the > >> current design, here are the directions that I would propose indeed: > >> > >> - a general fixed size list type. Don't do shortcuts just because the > >> discussion is too tedious. > >> - a vector type that indeed carries enough information to be really > >> useful for vector search by itself. (e.g., model name / version, > >> optional > >> guarantees for no 0 vector / normalized vector length). With such a > >> type, > >> you could really do well defined vector searches even if the writing > >> engine > >> is completely different to the reading one. > >> - Standardize quantizations. This will be a real game changer for > >> performance. > >> > >> Cheers, > >> Jan > >> > >> > >> > >> > >> > >> > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <[email protected]>: > >>> > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a > >> good > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more > natural. > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first? > >>> > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < > >>> [email protected]> wrote: > >>> > >>>> I suggest we go with VECTOR for the finite vector to keep the name the > >>> same > >>>> as other engines/type systems. If we want a non-finite one in the > >> future > >>> we > >>>> can name that something else or we add fixed size list that will serve > >>> the > >>>> same purpose without a special name. > >>>> > >>>> > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> wrote: > >>>> > >>>>> A logical type would: > >>>>> - Describe data for users and readers to understand what the file > >>>> contains > >>>>> - Signal that repetition and definition levels will reflect a fixed > >>>> number > >>>>> of elements, which implementations can use to optimize decoding > >>>>> - Guarantee that there are no null elements (because the type cannot > >> be > >>>>> applied to a list with optional elements) > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity > >> values > >>>> so > >>>>> that the vectors have a finite length and are suitable for distance > >>>>> calculations > >>>>> > >>>>> The finite value restriction is a restriction on writers to prevent > >> bad > >>>>> data from being written. > >>>>> > >>>>> Ignoring logical types that are not understood is a rule for > >> readers. A > >>>>> reader can handle vectors and pass it on to another layer. For > >>> instance, > >>>> a > >>>>> Parquet reader that doesn't support geometry passes values as > >> `binary`. > >>>>> That binary value can be passed to a geometry UDF, even if the > >>> underlying > >>>>> engine and Parquet reader doesn't have support. > >>>>> > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a > >> list > >>> of > >>>>> floats along to a UDF. But it is still important to the user that bad > >>>>> values are not present in the data. In your example, bad values are > >>>> vectors > >>>>> that cause the centroid calculation to fail, like vectors with > >> infinite > >>>>> length. > >>>>> > >>>>> Ryan > >>>>> > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < > >>>>> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> Could we clarify the use cases for a VECTOR annotation on > >>> LIST<FLOAT>? > >>>>>> Specifically, what does it make possible that's impossible today, > >> and > >>>>> what > >>>>>> does it make easier that's currently hard? > >>>>>> > >>>>>> I'm struggling to find a strong answer. I'd like every engine to > >>> offer > >>>>>> vector functions on lists of floats without needing a new logical > >>> type. > >>>>>> > >>>>>> To be clear, I'm still enthusiastic about a vector type that > >>> describes > >>>>> how > >>>>>> to unpack and scale composite representations (quantised codes plus > >>>>> scales, > >>>>>> codebooks, etc.), in the direction ML is going. That adds > >> information > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already > >>>>>> understand the physical type, and many already offer vector > >> functions > >>>> on > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are > >>>> cheap > >>>>>> enough that engines should do them anyway. The label also doesn't > >> say > >>>>> what > >>>>>> you can do with the column. Knowing that a column holds embeddings > >>>>> doesn't > >>>>>> say how they can be used, just as zipcode and age are both integers > >>> but > >>>>>> multiplying them is meaningless. > >>>>>> > >>>>>> My concern is the opposite effect. We're establishing that engines > >>>> should > >>>>>> ignore logical types they don't understand. So a tool that doesn't > >>> know > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute > >>>>> centroids > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines > >>> restrict > >>>>>> embedding functions to VECTOR-annotated columns, that file stops > >>>> working > >>>>>> with them, through no fault of the user. An annotation that doesn't > >>>>> change > >>>>>> what the data is shouldn't be able to make it unusable. > >>>>>> > >>>>>> Am I missing a use case? > >>>>>> > >>>>>> Best, Will > >>>>>> > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> > >> wrote: > >>>>>> > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many > >>> use > >>>>>> cases. > >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and > >> not > >>>> to > >>>>>> fit > >>>>>>> every use cases into the FIXED_LENGTH_LIST. > >>>>>>> > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> > >> wrote: > >>>>>>> > >>>>>>>> Going through this thread again, it looks like there are a few > >>>> things > >>>>>>> that > >>>>>>>> people generally agree on: > >>>>>>>> > >>>>>>>> - In the short term, we want to introduce a logical type or > >>> types; > >>>> a > >>>>>>> better > >>>>>>>> physical representation can be done later. We can focus on > >>> logical > >>>>>> types. > >>>>>>>> - There is precedent in logical types for restricting values, > >>> like > >>>>> UTF8 > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree > >>>> that > >>>>> we > >>>>>>>> could restrict vector elements to non-nested or primitive > >> types. > >>>>>>>> - There are many examples of databases that restrict their > >> vector > >>>>> types > >>>>>>> to > >>>>>>>> non-null, finite elements. > >>>>>>>> - Some people want to have write-time guarantees that vectors > >>> have > >>>>> only > >>>>>>>> non-null, finite elements > >>>>>>>> - Some people want to extend this work to include tensors, > >>> shorter > >>>>>>> floating > >>>>>>>> point representations, variable lengths, NaN or Infinity > >>> elements, > >>>>> etc. > >>>>>>>> > >>>>>>>> It sounds like we are nearly there, but have two halves of the > >>>>>> community > >>>>>>>> that care about different things. That just means we should > >> have > >>>>>> multiple > >>>>>>>> types. It's entirely reasonable to want a type that allows NaN > >>>>> values, > >>>>>>> and > >>>>>>>> it's also reasonable for those of us building databases, table > >>>>>> standards, > >>>>>>>> and vector indexes to care about guarantees that elements are > >>>> finite. > >>>>>>>> There's more than enough examples to show that this is a > >>> legitimate > >>>>> use > >>>>>>>> case. > >>>>>>>> > >>>>>>>> Since I don't see the concerns of the different use cases > >>> changing, > >>>>> it > >>>>>>>> looks like we need to find a compromise to move forward. I > >>> propose > >>>>> that > >>>>>>> we > >>>>>>>> introduce a FINITE_VECTOR type and separately a > >> FIXED_LENGTH_LIST > >>>> or > >>>>>>>> another VECTOR type that allows infinite and NaN values. > >>>>>>>> > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try > >> to > >>>> fit > >>>>>>> these > >>>>>>>> use cases into a single type? What if that means not making > >>>> progress? > >>>>>>>> > >>>>>>>> Ryan > >>>>>>>> > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < > >>>>>>>> [email protected]> wrote: > >>>>>>>> > >>>>>>>>> Thinking about this more, I am also fine with pulling out the > >>>>>> “NaN/Inf” > >>>>>>>>> requirement. Either into a type boolean (my preference), or > >>>> putting > >>>>>> no > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754 > >> float > >>>>>> numbers. > >>>>>>>> If > >>>>>>>>> some top-level application wants to put additional > >> constraints > >>> on > >>>>> the > >>>>>>>> data > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do > >>> so > >>>>> one > >>>>>>>> layer > >>>>>>>>> above. It seems like this was the biggest point of discussion > >>>>>>> remaining. > >>>>>>>>> All these options work for me, and we should move forward > >> with > >>>> this > >>>>>>>>> proposal. > >>>>>>>>> > >>>>>>>>> Concerning additional element types like bf16, they can be > >>>>> introduced > >>>>>>> as > >>>>>>>>> Parquet types (at least in the current LIST-based spec > >>> proposal). > >>>>>>>>> > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a > >>>>> variable > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I > >> think > >>>>> that > >>>>>> a > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits > >> by > >>>>>>>> describing > >>>>>>>>> the semantics of the data, and allowing to capture future > >>> changes > >>>>>>> around > >>>>>>>>> statistics. In that case, the length is better captured as an > >>>> exact > >>>>>>>> length > >>>>>>>>> for this type specifically. I’m also not sure about other use > >>>> cases > >>>>>> for > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just > >>> for > >>>>> the > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough > >>> value > >>>> in > >>>>>> my > >>>>>>>>> view. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Philipp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > >>>>>>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond > >> normalized > >>>>>>> embeddings > >>>>>>>>> and > >>>>>>>>>> engines really aren't going to be able to elide all the > >>> checks > >>>>>>> anyway. > >>>>>>>>>> > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :) > >>>>>>>>>> > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't > >>>> express > >>>>>> this > >>>>>>>> in > >>>>>>>>>> Parquet. Engines can write opaque blobs they understand in > >>> the > >>>>>>>> metadata > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but > >>> this > >>>>>> isn't > >>>>>>>>>> standardized, portable, nor interoperable. > >>>>>>>>>> > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift > >>>> fields > >>>>>> (my > >>>>>>>>> pref) > >>>>>>>>>> can generically support VECTOR(N) whilst also being more > >>>> broadly > >>>>>>> useful > >>>>>>>>> for > >>>>>>>>>> other use cases and data types? > >>>>>>>>>> > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and > >>>>>>> MAX_LENGTH > >>>>>>>>> for > >>>>>>>>>> Parquet's variable-length physcial types. These would > >> count > >>>>>> logical > >>>>>>>>> units > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes. > >>>>>>>>>> > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as > >>>> LIST(N) > >>>>>> with > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even > >> different > >>>>>>> engines - > >>>>>>>>> can > >>>>>>>>>> pick up these constraints if they support doing so. We > >> don't > >>>>>>> actually > >>>>>>>>> need > >>>>>>>>>> a logical type to express a VECTOR then. > >>>>>>>>>> > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to > >> be > >>>>>>> expressed > >>>>>>>> on > >>>>>>>>>> UTF8 columns etc. > >>>>>>>>>> > >>>>>>>>>> This also seems generally useful in the future as we > >> approach > >>>>>>>> expressing > >>>>>>>>>> other tensor data shapes and types. > >>>>>>>>>> > >>>>>>>>>> This won't change the mechanics: the storage type doesn't > >>>> enforce > >>>>>>> these > >>>>>>>>>> constraints and engines must check them. It's previously > >>>>>> established > >>>>>>>> that > >>>>>>>>>> performance is just a software problem and there is no > >>>>> fundamental > >>>>>>>>>> performance advantage to the fixed size types in > >> Dremel-like > >>>>>> formats > >>>>>>>> like > >>>>>>>>>> Parquet. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Will > >>>>>>>>>> > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > >>>>>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have > >>> been > >>>>>>>> thinking > >>>>>>>>>>> about this quite a bit, and thought I would share my 2 > >>> cents. > >>>>>>>>>>> > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as > >>>>> proposed > >>>>>>> in > >>>>>>>>> the > >>>>>>>>>> PR > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>. > >>>>>>>>>>> > >>>>>>>>>>> While Divjot and others note the specialization for > >> UTF-8, > >>>>>> Parquet > >>>>>>>>>> already > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings > >>>>>>>>>>> < > >>>>>>>> > >>>> https://github.com/apache/parquet-format/blob/master/README.md#types > >>>>>>>>>> . > >>>>>>>>>>> > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new > >>>>>>> restriction > >>>>>>>>> on > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE > >>>>> columns > >>>>>>>> have > >>>>>>>>>>> never had such a rule. > >>>>>>>>>>> > >>>>>>>>>>> It would also not match how the systems that actually > >>> produce > >>>>>> these > >>>>>>>>>> vectors > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark, > >>>> DuckDB, > >>>>>>>> NumPy, > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and > >>>> OpenBLAS. > >>>>>>>>>>> > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an > >> insert > >>>>>> policy. > >>>>>>>>> While > >>>>>>>>>> I > >>>>>>>>>>> think Parquet as a general-purpose format should allow > >> them > >>>> out > >>>>>> of > >>>>>>>> the > >>>>>>>>>> box. > >>>>>>>>>>> It is a user system decision to disallow writing these > >>> values > >>>>> or > >>>>>> to > >>>>>>>>> throw > >>>>>>>>>>> errors when encountering them. > >>>>>>>>>>> > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector > >>>> math. > >>>>>>>> Finite > >>>>>>>>>>> coordinates still overflow. For instance: magnitude = > >>>>>> ||[FLOAT_MAX, > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows > >> to > >>>>> +Inf. > >>>>>>>>> PyTorch > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm() > >> is > >>>> inf > >>>>>> in > >>>>>>>>> fp32. > >>>>>>>>>>> > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while > >>> the > >>>>>>> VECTOR > >>>>>>>>>> itself > >>>>>>>>>>> was required to be finite. The ban only forbids storing > >>>>> specials > >>>>>> in > >>>>>>>> the > >>>>>>>>>>> coordinates, and does not prevent even simple downstream > >>>>>>> computations > >>>>>>>>>> from > >>>>>>>>>>> producing them. > >>>>>>>>>>> > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional > >>> bool > >>>>>>>>>>> VectorType.finite (default false). Engines that want > >>>>>> pgvector-like > >>>>>>>>>>> rejection can require the flag = true. Engines that use > >>>> NumPy / > >>>>>>>>> PyTorch / > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the > >>> same > >>>>>>>> reason, I > >>>>>>>>>> do > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Sebastian > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > >>>>>>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I agree with moving forward with the proposal and > >>> renaming > >>>>>> VECTOR > >>>>>>>> to > >>>>>>>>>>>> FINITE_VECTOR or something > >>>>>>>>>>>> similar if others feel that better communicates its > >>>> intent. I > >>>>>> see > >>>>>>>>> value > >>>>>>>>>>> in > >>>>>>>>>>>> having a vector type that guarantees > >>>>>>>>>>>> its values are mathematically-correct vectors for a few > >>>>>> reasons: > >>>>>>>>>>>> > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the > >>> STRING > >>>>>>> logical > >>>>>>>>> type > >>>>>>>>>>>> mandating UTF-8. > >>>>>>>>>>>> 2. It is not possible to rely on statistics for > >> detecting > >>>> NaN > >>>>>> or > >>>>>>>>> +-inf > >>>>>>>>>>>> values as all statistics are optional. > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring > >> that > >>>>>> writers > >>>>>>>>> emit > >>>>>>>>>>>> stats for specific logical types. > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow > >> through > >>>> the > >>>>>>> engine > >>>>>>>>> and > >>>>>>>>>>>> block them for specific functionality. > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't > >> think > >>>>> this > >>>>>> is > >>>>>>>> as > >>>>>>>>>> easy > >>>>>>>>>>>> as it sounds and Lance has run > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples). > >>> Their > >>>>>>> Python > >>>>>>>>> [3] > >>>>>>>>>>> and > >>>>>>>>>>>> Rust [4] APIs default to erroring on > >>>>>>>>>>>> vectors that contain NaN values. > >>>>>>>>>>>> > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I > >>>> don't > >>>>>>> think > >>>>>>>>>> this > >>>>>>>>>>>> implies the lack of domain-specific types. > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet > >> would > >>> be > >>>>>>>>> considered > >>>>>>>>>> a > >>>>>>>>>>>> good fit. I also fully agree with > >>>>>>>>>>>> continuing work on a general fixed-size list type if we > >>>>> believe > >>>>>>>> there > >>>>>>>>>> are > >>>>>>>>>>>> non-vector use cases that would benefit > >>>>>>>>>>>> from such guarantees at the storage layer. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648 > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607 > >>> (see > >>>>>>> point 2 > >>>>>>>>> in > >>>>>>>>>>>> "root > >>>>>>>>>>>> cause") > >>>>>>>>>>>> [3] > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > >>>>>>>>>>>> [4] > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via > >>> dev < > >>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I also favor moving forward with the proposal as is. > >> I > >>>> see > >>>>>>> strong > >>>>>>>>>> value > >>>>>>>>>>>> and > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree > >> with > >>>> the > >>>>>>>>>>> restrictions > >>>>>>>>>>>>> around NaN/Inf values and element types in general. > >>>>> Precedent > >>>>>>>>> exists > >>>>>>>>>> in > >>>>>>>>>>>>> other types that restrict which physical type values > >>> are > >>>>>>>> logically > >>>>>>>>>>> valid > >>>>>>>>>>>>> (e.g., the geospatial types needing to be > >> WKB-encoded). > >>>> A > >>>>>>>> general > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes > >>>> sense > >>>>> to > >>>>>>>>>> separate > >>>>>>>>>>>> the > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see > >>> how > >>>>> the > >>>>>>> two > >>>>>>>>>> types > >>>>>>>>>>>>> might diverge further in the future regarding > >>> statistics > >>>> or > >>>>>>>>>> encodings. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it > >>>> seems > >>>>>>> like > >>>>>>>>> the > >>>>>>>>>>>> VECTOR > >>>>>>>>>>>>> name is already used across other engines mentioned > >> in > >>>> this > >>>>>>>> thread. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Philipp > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < > >>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should > >>> treat > >>>>> the > >>>>>>>>> general > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout, > >> and > >>>>>> separate > >>>>>>>> it > >>>>>>>>>> from > >>>>>>>>>>>> the > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML > >>>> tensor) > >>>>>>>> logical > >>>>>>>>>>> type. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Currently, the immediate need is to define a > >>>> specialized > >>>>>>> VECTOR > >>>>>>>>>> type > >>>>>>>>>>> so > >>>>>>>>>>>>>> that different engines and formats have a common > >>>> semantic > >>>>>>>>> contract > >>>>>>>>>>> for > >>>>>>>>>>>>>> interoperability. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If the main point of contention is just the > >>>> NaN/Inf/null > >>>>>>>>>> limitations, > >>>>>>>>>>>>> maybe > >>>>>>>>>>>>>> we can resolve this by adding a qualification to > >> the > >>>> name > >>>>>>> (like > >>>>>>>>>>> Andrew > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) > >>> to > >>>>>> avoid > >>>>>>>>>>> confusion > >>>>>>>>>>>>> with > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could > >>>> introduce a > >>>>>>>> boolean > >>>>>>>>>>> flag > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed > >>>>>> earlier. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > >>>>>>>>>> [email protected]> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I am a little confused about the points of > >>>> disagreement > >>>>>>> here. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for > >> the > >>>>>>> specific > >>>>>>>>> use > >>>>>>>>>>> case > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used > >> for > >>>>>> vector > >>>>>>>>>>>> embeddings? > >>>>>>>>>>>>>> If > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the > >> name > >>>>>>>>>> (FINITE_VECTOR? > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It seems like there is precedent for defining > >>> logical > >>>>>> types > >>>>>>>>> that > >>>>>>>>>>>>> specify > >>>>>>>>>>>>>>> certain physical types and restrict their > >> contents > >>> to > >>>>>> help > >>>>>>>>>> readers > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON; > >>>>>> Defining > >>>>>>>> one > >>>>>>>>>> for > >>>>>>>>>>>>>> vector > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [1]: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > >>>>>>>>> [email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits > >>>> their > >>>>>>>> usage, > >>>>>>>>>> but > >>>>>>>>>>>>> we're > >>>>>>>>>>>>>>>>> talking about a general-purpose file format > >>> that > >>>>> aims > >>>>>>> to > >>>>>>>> be > >>>>>>>>>>>> broadly > >>>>>>>>>>>>>>>>> applicable. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I agree that the format is intended to be > >> general > >>>>>>> purpose, > >>>>>>>>> but > >>>>>>>>>>>> types > >>>>>>>>>>>>>>> within > >>>>>>>>>>>>>>>> the format define specific representation and > >>>>> semantics > >>>>>>> to > >>>>>>>>>>> achieve > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> goal. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > >>>> it's > >>>>>>> quite > >>>>>>>>>>> trivial > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > >>> e.g. > >>>>>> NumPy > >>>>>>>> or > >>>>>>>>>>> Pandas > >>>>>>>>>>>>> (or > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > >>>>> computing > >>>>>>>>>> software). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The justification for restricting non-finite > >>>> values > >>>>> is > >>>>>>>>>> directly > >>>>>>>>>>>>>> related > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> the semantic meaning of a vector. Vectors have > >>>>>> specific > >>>>>>>>>>>>>> characteristics > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). > >>> Non-finite > >>>>>>> values > >>>>>>>>> are > >>>>>>>>>>>>> invalid > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> break those characteristics. I'd liken it to > >>>> storing > >>>>>>>>> non-utf8 > >>>>>>>>>>>> values > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>> string and still calling it a string. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed > >>>> length > >>>>>>> list, > >>>>>>>>>> which > >>>>>>>>>>>> is > >>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>> general and does not carry the same semantic > >>>> meaning. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I also don't think we need to bundle all > >>> similarly > >>>>>> shaped > >>>>>>>> use > >>>>>>>>>>> cases > >>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list). > >> Vector > >>>> is > >>>>> a > >>>>>>> well > >>>>>>>>>>>>> established > >>>>>>>>>>>>>>>> data type at this point and aligns with the > >>> general > >>>>>>> purpose > >>>>>>>>>>> nature > >>>>>>>>>>>> of > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> format (many other formats and databases > >> natively > >>>>>> support > >>>>>>>>>> this). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -Dan > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine > >> Pitrou < > >>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a > >> écrit > >>> : > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the > >>>> clear > >>>>>>>>> use-case > >>>>>>>>>>> for > >>>>>>>>>>>>>>>> non-finite > >>>>>>>>>>>>>>>>>> values within a vector type. There has > >> been a > >>>> lot > >>>>>> of > >>>>>>>>>>>> theoretical > >>>>>>>>>>>>>>>>> discussion > >>>>>>>>>>>>>>>>>> here about use cases but what toolings > >>>> currently > >>>>>>>> produce > >>>>>>>>>>> these > >>>>>>>>>>>>>>> vectors > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> how do they function in downstream engines? > >>> I'd > >>>>>> like > >>>>>>> to > >>>>>>>>>> limit > >>>>>>>>>>>>>>>> introducing > >>>>>>>>>>>>>>>>>> types that have a wide range of values > >> which > >>>> are > >>>>>>> simply > >>>>>>>>> are > >>>>>>>>>>>>> failure > >>>>>>>>>>>>>>>> modes > >>>>>>>>>>>>>>>>>> of different flavors > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > >>>> it's > >>>>>>> quite > >>>>>>>>>>> trivial > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > >>> e.g. > >>>>>> NumPy > >>>>>>>> or > >>>>>>>>>>> Pandas > >>>>>>>>>>>>> (or > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > >>>>> computing > >>>>>>>>>> software). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector > >>> is > >>>> a > >>>>>> IETF > >>>>>>>>>>> standard > >>>>>>>>>>>> or > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8 > >> (where > >>>> the > >>>>>>>> values > >>>>>>>>>> are > >>>>>>>>>>>>>> expected > >>>>>>>>>>>>>>>>> to be valid according to the respective > >> specs) > >>> do > >>>>> not > >>>>>>>>> apply. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better > >> chance > >>>> of > >>>>>>>>> converging > >>>>>>>>>>> if > >>>>>>>>>>>> we > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data > >>> in > >>>>>>> Parquet > >>>>>>>>>> (with > >>>>>>>>>>> a > >>>>>>>>>>>>>>>>> potential limitation to non-nested types) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. representing the information that a given > >>>> column > >>>>>> (or > >>>>>>>>>> column > >>>>>>>>>>>>> chunk, > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ... both of which are useful on their own > >>> right, > >>>>> but > >>>>>> do > >>>>>>>> not > >>>>>>>>>>> have > >>>>>>>>>>>> to > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>> conflated and bundled together. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Regards > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Antoine. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
