> > > Daniel, alright, so no physical optimizations intended. We only want this for semantic reasons, not for any significant physical optimization. If we're all on the same page here, that is good ð.
To phrase my remaining problem with this more concretely: *Let's say we add this type. What cool new stuff can we now do that we couldn't do before?* That is the litmus test that a type has to pass to be useful IMHO. And here I can't come up with a tangible use case that wasn't possible before. Especially if we one day have fixed size lists in Parquet, the remaining niche for this type becomes so small, that the added complexity isn't worth the benefit IMHO. As I guess most people on here are active maintainers of a Parquet implementation: Concretely, what would you be using this type for if we had it? Would you allow math only on fields tagged with this type? What if that math yields INF or NaN? Not allowed? Especially if we also discuss having both VECTOR and FINITE_VECTOR. How will you guys wire this up with the type systems of your engines? Do your engines also have two types here? Do you type-decay a FINITE_VECTOR to a VECTOR once someone does a multiplication that could yield INF? By itself, a FINITE_VECTOR sounds good. Holistically looked at from a type system perspective, it comes with a lot of complexity. Do the benefits really outweigh the complexity here? That's why I would rather have the bare minimum: A fixed size array (list) of floats or at least just a vector without finite. Then all these detailed questions go away. I'll take this one step further... Why is there any debate about this at > all? Why can't a type simply be added to the *specification* as an > administrative detail and be done? Andrew, Parquet is supposed to be a lingua franca of data. The specification is the authoritative truth for this language and all implementations are expected to implement as much as possible from this spec, so anything in the spec does causes implementation effort. So just adding anything to this spec without debate is not a good idea. It is even the other way around: We need to have way higher rigor with the spec than we need to have with any implementations. Implementations can be changed, the spec needs to be stable. A tag-only type that doesn't come with any restrictions can be added with little effort. But here we want to tie restrictions to it (no NaN/Inf), so we would expect implementations to enforce these. So there is some implementation effort expected here. Also, even if we only wanted a tag type, I would still argue that you shouldn't add those without a debate. What if A wants XML, B wants YAML, C wants quaternions, D wants address, E wants phone number. If we just added any types that are "just tags", Parquet logical types would just become a trash dump of any type that anyone ever thought to be useful. So rigorous assessment of each new type candidate is necessary. Finally, many engines couple parquet with another type system (e.g., Arrow or SQL). So even with a tag only type, you at least still have the implementation burden of wiring this up with your engine's type system. So, just because logical types are "just" logical, that doesn't mean we can add them without any rigor. A few naive questions: > 1. Would a physical definition with NANs & Inf+/- allow efficient > encodings like ALP? > 2. If the physical representation allows values that make encodings like > ALP impossible, can a tighter Logical type allow different encodings? > 3. Can we RLE or binpack a vector? The simple general answer here is that all these things do not require a logical type. Writers can already do any such optimization if the actual data contains no NaNs & Infs. All these physical optimizations just require no actual NaNs/Infs to be present in a page, they don't need a guarantee that the whole file won't contain any. Cheers, Jan Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <[email protected]>: > Hi Everyone, > > Glad to see a Vector type being added. I have a few use cases I want to > pursue once we have an agreed upon definition. > > In regards to that definition discussion. I feel comfortable with the idea > of the semantic meaning Dan raises though I canât shake a slight discomfort > (maybe FOMO) with limiting values in the physical definition. > > A few naive questions: > 1. Would a physical definition with NANs & Inf+/- allow efficient > encodings like ALP? > 2. If the physical representation allows values that make encodings like > ALP impossible, can a tighter Logical type allow different encodings? > 3. Can we RLE or binpack a vector? > > Best Regards, > Kurtis C. Wright > > On Oct 14, 2026, at 12:45, Daniel Weeks <[email protected]> wrote: > > > > Hey Jan, > > > > There's a lot that I disagree with respect to the framing of these > > arguments and how we should be thinking about introducing vector as a > > logical type. There's a repeated tendency to let the physical > > representation or performance drive discussion around types as opposed to > > what it represents. > > > > In parquet, establishing semantic meaning is the purpose of logical > types. > > The logical type separates how to interpret a value from its physical > > representation. We can represent a timestamp with INT64 or even a string > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and > VECTOR > > may have the exact same representation and performance, they have > different > > semantic meaning. Databases and engines across the ecosystem already > > recognize this differentiation and have separate representations. The > > choice to separate these isn't because the discussion is too tedious; > > they're fundamentally different things. > > > > The finite requirement is what makes a vector a reflection of what it > > represents. As we've discussed in the thread, non-finite values in the > > representation violate the properties of a vector and the value ceases to > > be a vector. On the other hand, a 0 vector is still a vector (you can > > argue the utility of it, but it is a valid vector). Implementations can > > choose what they validate or how to handle execution errors; that's an > > implementation concern, not a requirement. Operations performed on a > > vector depend on the context in which they are used, but the > representation > > doesn't need to solve scenarios where that application doesn't make > sense. > > > > The guarantees or context needed to use a vector do not need to be > inherent > > in the type for it to be a useful type. We don't need the currency used > > for decimal or whether a floating point is a normalized value as part of > > the type to make use of those types. This conflates the use cases with > the > > type system and unnecessarily attempts to draw the context into the type > > definition. Similarly, you don't need to capture normalization, model, > > etc. as part of the type for it to be useful. > > > > So the reason we want to add a vector type (or FINITE_VECTOR) is because > we > > want to store vectors with the semantic meaning and requirements based on > > what they represent. > > > > -Dan > > > > > >> On Wed, Oct 14, 2026 at 1:22â¯AM Jan Finis <[email protected]> wrote: > >> > >> A bit late to the party, but here we go: > >> > >> I have the strong sentiment that the FINITE_VECTOR type as it is > currently > >> being discussed will not be very useful and just increase implementation > >> complexity. The arguments for the type are all quite weak (details see > >> below). The FINITE part doesn't seem to yield any value and is not a > useful > >> restriction by itself. The vector (i.e. fixed length) part should > rather be > >> generalized instead of making a one-off thing. In general, the existing > >> types are already working well for uncompressed vectors. The real > challenge > >> that would bring real benefits is supporting quantization. This is where > >> you get orders of magnitude improvements to storage space and processing > >> speed. > >> > >> Let's first discuss the FINITE part. So far, the only arguments I have > read > >> for this are: > >> > >> 1. many other vector databases have it > >> 2. vectors are the hot stuff right now, so Parquet better boast having > >> "native" type support for vectors > >> 3. by focussing on this common special case, we can make rapid progress, > >> while a general solution would take too long to specify > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the > >> vector > >> > >> the first two arguments are quite weak by themselves. Vector databases > are > >> very specialized engines. Such engines can of course be very > opinionated on > >> their data types. A general format like Parquet must instead weigh > whether > >> a special type is worth the added complexity over a more general, widely > >> applicable type. In this case, it is IMHO not at all. And just having a > >> type called "vector" for advertisement reasons to make Parquet appear > more > >> modern seems a non-goal in the first place. > >> > >> The third "time to market" argument is valid for systems that don't > need a > >> stable interface. It is a very dangerous attitude for formats like > Parquet > >> that must be stable long term and should have a consolidated, concise, > >> non-redundant type system. For such long term storage formats, we need > to > >> take the time to build the right thing, as any shortcuts we take will > stay > >> as blemishes in the format forever. If I read the discussion correctly, > >> this is actually the main argument as of now: We have found that it is > hard > >> to align on how a fixed size list type should look like, so we want to > go > >> the easier route of a specialized type. This seems like a > >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would > we > >> want this, if Parquet already had a fixed size list type? > >> > >> So finally, the only technical argument is the fourth one: "It signals > >> semantics and ensures that no bad data lands in the format". I see two > big > >> problems with this: > >> > >> 1. Readers *still* need to test for NaN / INF because Parquet files may > be > >> corrupt. Nothing can stop a malicious or buggy writer from physically > >> writing NaN or INF. *So you save no single line of code in a reader. You > >> save no branches, as you still need to handle the INF/NaN case.* So this > >> doesn't help with performance at all. The only difference is, that the > >> error message changes from "found an invalid INF value in the data" to > >> "Your parquet file seems to be corrupt". It is actually quite the > opposite, > >> readers and writers get more complex having to handle this new type. > >> > >> If you now argue that it's not about performance, it's about defined > >> semantics and erroring out early in writers, then the second problem > >> arises: > >> > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple > >> guarantees that you need for sensible vector processing. If "no > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that > >> FINITE_VECTOR is useful, but it is by no means the only guarantee you > need, > >> instead, depending on the distance metric, you also need the following > >> guarantees. E.g.: > >> > >> - No vector may be the 0 vector, otherwise, e.g., cosine similarity is > >> undefined. Cosine similarity is often used as default, so this is > >> actually > >> quite important in practice. > >> - If we want to use a dot product, then all vectors need to guarantee > a > >> length of 1. This would actually be quite useful for performance > >> reasons. > >> - The vectors must come from the same model as the vector used for the > >> search. Otherwise, the search yields a numeric result, but that result > >> has > >> no sensible meaning. > >> > >> So, what use is a type that just guarantees part of the things you need > for > >> vector search but not others? You still need additional guarantees in > >> addition to that type. What makes INF/NaN so important in comparison to, > >> say, "no 0 vector present", that it warrants an own type, while the > other > >> guarantees do not? If we really want this type as a basis for > cross-engine > >> sharing of vectors, then it needs to give *all* guarantees. > >> > >> I'm not against a specialized vector type if it really carries *all* > >> information to be useful by itself. If it carried all that information, > >> then a reading engine could indeed validate that only meaningful vector > >> searches are executed. But as long as it is just a half-baked thing, it > is > >> not useful by itself and just a burden on each implementer and > maintainer. > >> If we need extra information anyway, why not just stick to a general > type > >> like VECTOR (without FINITE). > >> > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what > >> advantage you will *really* get over just a (potentially fixed size) > List > >> of Float? As discussed, it's not performance. It's also not a semantic > >> marker that is sufficient by itself. It's just a small hint that allows > >> writers to error out once they see a NaN or INF in the input, while they > >> will still happily write a bogus 0 vector. This seems to be of zero > >> benefit. > >> > >> Anecdotally, we currently handle petabytes of vectors in production in > >> Data360. We simply store them in a List of Float, with an optimized > R-Level > >> reader implementation that rapidly checks that all vectors are the same > >> length. This solution works just fine. The overhead of R-level decoding > is > >> barely measurable. The check is super fast and R-levels are tiny in > >> comparison to the payload vectors, so also the added I/O is negligible. > >> This is also why I'm so firm in my sentiment: We are running a > production > >> grade system processing trillions of vectors right now and see no big > >> issues with the types that Parquet currently provides. > >> > >> As I don't just want to be a naysayer and point out problems with the > >> current design, here are the directions that I would propose indeed: > >> > >> - a general fixed size list type. Don't do shortcuts just because the > >> discussion is too tedious. > >> - a vector type that indeed carries enough information to be really > >> useful for vector search by itself. (e.g., model name / version, > >> optional > >> guarantees for no 0 vector / normalized vector length). With such a > >> type, > >> you could really do well defined vector searches even if the writing > >> engine > >> is completely different to the reading one. > >> - Standardize quantizations. This will be a real game changer for > >> performance. > >> > >> Cheers, > >> Jan > >> > >> > >> > >> > >> > >> > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <[email protected]>: > >>> > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a > >> good > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more > natural. > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first? > >>> > >>> On Tue, Oct 13, 2026 at 6:26â¯PM Alkis Evlogimenos via dev < > >>> [email protected]> wrote: > >>> > >>>> I suggest we go with VECTOR for the finite vector to keep the name the > >>> same > >>>> as other engines/type systems. If we want a non-finite one in the > >> future > >>> we > >>>> can name that something else or we add fixed size list that will serve > >>> the > >>>> same purpose without a special name. > >>>> > >>>> > >>>> On Thu, Oct 8, 2026 at 10:20â¯PM Ryan Blue <[email protected]> wrote: > >>>> > >>>>> A logical type would: > >>>>> - Describe data for users and readers to understand what the file > >>>> contains > >>>>> - Signal that repetition and definition levels will reflect a fixed > >>>> number > >>>>> of elements, which implementations can use to optimize decoding > >>>>> - Guarantee that there are no null elements (because the type cannot > >> be > >>>>> applied to a list with optional elements) > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity > >> values > >>>> so > >>>>> that the vectors have a finite length and are suitable for distance > >>>>> calculations > >>>>> > >>>>> The finite value restriction is a restriction on writers to prevent > >> bad > >>>>> data from being written. > >>>>> > >>>>> Ignoring logical types that are not understood is a rule for > >> readers. A > >>>>> reader can handle vectors and pass it on to another layer. For > >>> instance, > >>>> a > >>>>> Parquet reader that doesn't support geometry passes values as > >> `binary`. > >>>>> That binary value can be passed to a geometry UDF, even if the > >>> underlying > >>>>> engine and Parquet reader doesn't have support. > >>>>> > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a > >> list > >>> of > >>>>> floats along to a UDF. But it is still important to the user that bad > >>>>> values are not present in the data. In your example, bad values are > >>>> vectors > >>>>> that cause the centroid calculation to fail, like vectors with > >> infinite > >>>>> length. > >>>>> > >>>>> Ryan > >>>>> > >>>>> On Thu, Oct 8, 2026 at 12:54â¯PM Will Edwards via dev < > >>>>> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> Could we clarify the use cases for a VECTOR annotation on > >>> LIST<FLOAT>? > >>>>>> Specifically, what does it make possible that's impossible today, > >> and > >>>>> what > >>>>>> does it make easier that's currently hard? > >>>>>> > >>>>>> I'm struggling to find a strong answer. I'd like every engine to > >>> offer > >>>>>> vector functions on lists of floats without needing a new logical > >>> type. > >>>>>> > >>>>>> To be clear, I'm still enthusiastic about a vector type that > >>> describes > >>>>> how > >>>>>> to unpack and scale composite representations (quantised codes plus > >>>>> scales, > >>>>>> codebooks, etc.), in the direction ML is going. That adds > >> information > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already > >>>>>> understand the physical type, and many already offer vector > >> functions > >>>> on > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are > >>>> cheap > >>>>>> enough that engines should do them anyway. The label also doesn't > >> say > >>>>> what > >>>>>> you can do with the column. Knowing that a column holds embeddings > >>>>> doesn't > >>>>>> say how they can be used, just as zipcode and age are both integers > >>> but > >>>>>> multiplying them is meaningless. > >>>>>> > >>>>>> My concern is the opposite effect. We're establishing that engines > >>>> should > >>>>>> ignore logical types they don't understand. So a tool that doesn't > >>> know > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute > >>>>> centroids > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines > >>> restrict > >>>>>> embedding functions to VECTOR-annotated columns, that file stops > >>>> working > >>>>>> with them, through no fault of the user. An annotation that doesn't > >>>>> change > >>>>>> what the data is shouldn't be able to make it unusable. > >>>>>> > >>>>>> Am I missing a use case? > >>>>>> > >>>>>> Best, Will > >>>>>> > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> > >> wrote: > >>>>>> > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many > >>> use > >>>>>> cases. > >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and > >> not > >>>> to > >>>>>> fit > >>>>>>> every use cases into the FIXED_LENGTH_LIST. > >>>>>>> > >>>>>>> On Mon, Oct 5, 2026 at 1:26â¯PM Ryan Blue <[email protected]> > >> wrote: > >>>>>>> > >>>>>>>> Going through this thread again, it looks like there are a few > >>>> things > >>>>>>> that > >>>>>>>> people generally agree on: > >>>>>>>> > >>>>>>>> - In the short term, we want to introduce a logical type or > >>> types; > >>>> a > >>>>>>> better > >>>>>>>> physical representation can be done later. We can focus on > >>> logical > >>>>>> types. > >>>>>>>> - There is precedent in logical types for restricting values, > >>> like > >>>>> UTF8 > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree > >>>> that > >>>>> we > >>>>>>>> could restrict vector elements to non-nested or primitive > >> types. > >>>>>>>> - There are many examples of databases that restrict their > >> vector > >>>>> types > >>>>>>> to > >>>>>>>> non-null, finite elements. > >>>>>>>> - Some people want to have write-time guarantees that vectors > >>> have > >>>>> only > >>>>>>>> non-null, finite elements > >>>>>>>> - Some people want to extend this work to include tensors, > >>> shorter > >>>>>>> floating > >>>>>>>> point representations, variable lengths, NaN or Infinity > >>> elements, > >>>>> etc. > >>>>>>>> > >>>>>>>> It sounds like we are nearly there, but have two halves of the > >>>>>> community > >>>>>>>> that care about different things. That just means we should > >> have > >>>>>> multiple > >>>>>>>> types. It's entirely reasonable to want a type that allows NaN > >>>>> values, > >>>>>>> and > >>>>>>>> it's also reasonable for those of us building databases, table > >>>>>> standards, > >>>>>>>> and vector indexes to care about guarantees that elements are > >>>> finite. > >>>>>>>> There's more than enough examples to show that this is a > >>> legitimate > >>>>> use > >>>>>>>> case. > >>>>>>>> > >>>>>>>> Since I don't see the concerns of the different use cases > >>> changing, > >>>>> it > >>>>>>>> looks like we need to find a compromise to move forward. I > >>> propose > >>>>> that > >>>>>>> we > >>>>>>>> introduce a FINITE_VECTOR type and separately a > >> FIXED_LENGTH_LIST > >>>> or > >>>>>>>> another VECTOR type that allows infinite and NaN values. > >>>>>>>> > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try > >> to > >>>> fit > >>>>>>> these > >>>>>>>> use cases into a single type? What if that means not making > >>>> progress? > >>>>>>>> > >>>>>>>> Ryan > >>>>>>>> > >>>>>>>> On Mon, Oct 5, 2026 at 12:15â¯PM Philipp Fischbeck via dev < > >>>>>>>> [email protected]> wrote: > >>>>>>>> > >>>>>>>>> Thinking about this more, I am also fine with pulling out the > >>>>>> âNaN/Infâ > >>>>>>>>> requirement. Either into a type boolean (my preference), or > >>>> putting > >>>>>> no > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754 > >> float > >>>>>> numbers. > >>>>>>>> If > >>>>>>>>> some top-level application wants to put additional > >> constraints > >>> on > >>>>> the > >>>>>>>> data > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do > >>> so > >>>>> one > >>>>>>>> layer > >>>>>>>>> above. It seems like this was the biggest point of discussion > >>>>>>> remaining. > >>>>>>>>> All these options work for me, and we should move forward > >> with > >>>> this > >>>>>>>>> proposal. > >>>>>>>>> > >>>>>>>>> Concerning additional element types like bf16, they can be > >>>>> introduced > >>>>>>> as > >>>>>>>>> Parquet types (at least in the current LIST-based spec > >>> proposal). > >>>>>>>>> > >>>>>>>>> Concerning Willâs suggestion around representing VECTOR as a > >>>>> variable > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I > >> think > >>>>> that > >>>>>> a > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits > >> by > >>>>>>>> describing > >>>>>>>>> the semantics of the data, and allowing to capture future > >>> changes > >>>>>>> around > >>>>>>>>> statistics. In that case, the length is better captured as an > >>>> exact > >>>>>>>> length > >>>>>>>>> for this type specifically. Iâm also not sure about other use > >>>> cases > >>>>>> for > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just > >>> for > >>>>> the > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesnât have enough > >>> value > >>>> in > >>>>>> my > >>>>>>>>> view. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Philipp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00â¯AM Will Edwards via dev < > >>>>>>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond > >> normalized > >>>>>>> embeddings > >>>>>>>>> and > >>>>>>>>>> engines really aren't going to be able to elide all the > >>> checks > >>>>>>> anyway. > >>>>>>>>>> > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :) > >>>>>>>>>> > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't > >>>> express > >>>>>> this > >>>>>>>> in > >>>>>>>>>> Parquet. Engines can write opaque blobs they understand in > >>> the > >>>>>>>> metadata > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but > >>> this > >>>>>> isn't > >>>>>>>>>> standardized, portable, nor interoperable. > >>>>>>>>>> > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift > >>>> fields > >>>>>> (my > >>>>>>>>> pref) > >>>>>>>>>> can generically support VECTOR(N) whilst also being more > >>>> broadly > >>>>>>> useful > >>>>>>>>> for > >>>>>>>>>> other use cases and data types? > >>>>>>>>>> > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and > >>>>>>> MAX_LENGTH > >>>>>>>>> for > >>>>>>>>>> Parquet's variable-length physcial types. These would > >> count > >>>>>> logical > >>>>>>>>> units > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes. > >>>>>>>>>> > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as > >>>> LIST(N) > >>>>>> with > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even > >> different > >>>>>>> engines - > >>>>>>>>> can > >>>>>>>>>> pick up these constraints if they support doing so. We > >> don't > >>>>>>> actually > >>>>>>>>> need > >>>>>>>>>> a logical type to express a VECTOR then. > >>>>>>>>>> > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to > >> be > >>>>>>> expressed > >>>>>>>> on > >>>>>>>>>> UTF8 columns etc. > >>>>>>>>>> > >>>>>>>>>> This also seems generally useful in the future as we > >> approach > >>>>>>>> expressing > >>>>>>>>>> other tensor data shapes and types. > >>>>>>>>>> > >>>>>>>>>> This won't change the mechanics: the storage type doesn't > >>>> enforce > >>>>>>> these > >>>>>>>>>> constraints and engines must check them. It's previously > >>>>>> established > >>>>>>>> that > >>>>>>>>>> performance is just a software problem and there is no > >>>>> fundamental > >>>>>>>>>> performance advantage to the fixed size types in > >> Dremel-like > >>>>>> formats > >>>>>>>> like > >>>>>>>>>> Parquet. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Will > >>>>>>>>>> > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > >>>>>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have > >>> been > >>>>>>>> thinking > >>>>>>>>>>> about this quite a bit, and thought I would share my 2 > >>> cents. > >>>>>>>>>>> > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as > >>>>> proposed > >>>>>>> in > >>>>>>>>> the > >>>>>>>>>> PR > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>. > >>>>>>>>>>> > >>>>>>>>>>> While Divjot and others note the specialization for > >> UTF-8, > >>>>>> Parquet > >>>>>>>>>> already > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings > >>>>>>>>>>> < > >>>>>>>> > >>>> https://github.com/apache/parquet-format/blob/master/README.md#types > >>>>>>>>>> . > >>>>>>>>>>> > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new > >>>>>>> restriction > >>>>>>>>> on > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE > >>>>> columns > >>>>>>>> have > >>>>>>>>>>> never had such a rule. > >>>>>>>>>>> > >>>>>>>>>>> It would also not match how the systems that actually > >>> produce > >>>>>> these > >>>>>>>>>> vectors > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark, > >>>> DuckDB, > >>>>>>>> NumPy, > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and > >>>> OpenBLAS. > >>>>>>>>>>> > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an > >> insert > >>>>>> policy. > >>>>>>>>> While > >>>>>>>>>> I > >>>>>>>>>>> think Parquet as a general-purpose format should allow > >> them > >>>> out > >>>>>> of > >>>>>>>> the > >>>>>>>>>> box. > >>>>>>>>>>> It is a user system decision to disallow writing these > >>> values > >>>>> or > >>>>>> to > >>>>>>>>> throw > >>>>>>>>>>> errors when encountering them. > >>>>>>>>>>> > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector > >>>> math. > >>>>>>>> Finite > >>>>>>>>>>> coordinates still overflow. For instance: magnitude = > >>>>>> ||[FLOAT_MAX, > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows > >> to > >>>>> +Inf. > >>>>>>>>> PyTorch > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm() > >> is > >>>> inf > >>>>>> in > >>>>>>>>> fp32. > >>>>>>>>>>> > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while > >>> the > >>>>>>> VECTOR > >>>>>>>>>> itself > >>>>>>>>>>> was required to be finite. The ban only forbids storing > >>>>> specials > >>>>>> in > >>>>>>>> the > >>>>>>>>>>> coordinates, and does not prevent even simple downstream > >>>>>>> computations > >>>>>>>>>> from > >>>>>>>>>>> producing them. > >>>>>>>>>>> > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional > >>> bool > >>>>>>>>>>> VectorType.finite (default false). Engines that want > >>>>>> pgvector-like > >>>>>>>>>>> rejection can require the flag = true. Engines that use > >>>> NumPy / > >>>>>>>>> PyTorch / > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the > >>> same > >>>>>>>> reason, I > >>>>>>>>>> do > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Sebastian > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03â¯PM Divjot Arora via dev < > >>>>>>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I agree with moving forward with the proposal and > >>> renaming > >>>>>> VECTOR > >>>>>>>> to > >>>>>>>>>>>> FINITE_VECTOR or something > >>>>>>>>>>>> similar if others feel that better communicates its > >>>> intent. I > >>>>>> see > >>>>>>>>> value > >>>>>>>>>>> in > >>>>>>>>>>>> having a vector type that guarantees > >>>>>>>>>>>> its values are mathematically-correct vectors for a few > >>>>>> reasons: > >>>>>>>>>>>> > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the > >>> STRING > >>>>>>> logical > >>>>>>>>> type > >>>>>>>>>>>> mandating UTF-8. > >>>>>>>>>>>> 2. It is not possible to rely on statistics for > >> detecting > >>>> NaN > >>>>>> or > >>>>>>>>> +-inf > >>>>>>>>>>>> values as all statistics are optional. > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring > >> that > >>>>>> writers > >>>>>>>>> emit > >>>>>>>>>>>> stats for specific logical types. > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow > >> through > >>>> the > >>>>>>> engine > >>>>>>>>> and > >>>>>>>>>>>> block them for specific functionality. > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't > >> think > >>>>> this > >>>>>> is > >>>>>>>> as > >>>>>>>>>> easy > >>>>>>>>>>>> as it sounds and Lance has run > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples). > >>> Their > >>>>>>> Python > >>>>>>>>> [3] > >>>>>>>>>>> and > >>>>>>>>>>>> Rust [4] APIs default to erroring on > >>>>>>>>>>>> vectors that contain NaN values. > >>>>>>>>>>>> > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I > >>>> don't > >>>>>>> think > >>>>>>>>>> this > >>>>>>>>>>>> implies the lack of domain-specific types. > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet > >> would > >>> be > >>>>>>>>> considered > >>>>>>>>>> a > >>>>>>>>>>>> good fit. I also fully agree with > >>>>>>>>>>>> continuing work on a general fixed-size list type if we > >>>>> believe > >>>>>>>> there > >>>>>>>>>> are > >>>>>>>>>>>> non-vector use cases that would benefit > >>>>>>>>>>>> from such guarantees at the storage layer. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648 > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607 > >>> (see > >>>>>>> point 2 > >>>>>>>>> in > >>>>>>>>>>>> "root > >>>>>>>>>>>> cause") > >>>>>>>>>>>> [3] > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > >>>>>>>>>>>> [4] > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19â¯PM Philipp Fischbeck via > >>> dev < > >>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I also favor moving forward with the proposal as is. > >> I > >>>> see > >>>>>>> strong > >>>>>>>>>> value > >>>>>>>>>>>> and > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree > >> with > >>>> the > >>>>>>>>>>> restrictions > >>>>>>>>>>>>> around NaN/Inf values and element types in general. > >>>>> Precedent > >>>>>>>>> exists > >>>>>>>>>> in > >>>>>>>>>>>>> other types that restrict which physical type values > >>> are > >>>>>>>> logically > >>>>>>>>>>> valid > >>>>>>>>>>>>> (e.g., the geospatial types needing to be > >> WKB-encoded). > >>>> A > >>>>>>>> general > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes > >>>> sense > >>>>> to > >>>>>>>>>> separate > >>>>>>>>>>>> the > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see > >>> how > >>>>> the > >>>>>>> two > >>>>>>>>>> types > >>>>>>>>>>>>> might diverge further in the future regarding > >>> statistics > >>>> or > >>>>>>>>>> encodings. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it > >>>> seems > >>>>>>> like > >>>>>>>>> the > >>>>>>>>>>>> VECTOR > >>>>>>>>>>>>> name is already used across other engines mentioned > >> in > >>>> this > >>>>>>>> thread. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Philipp > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37â¯PM Gang Wu < > >>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should > >>> treat > >>>>> the > >>>>>>>>> general > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout, > >> and > >>>>>> separate > >>>>>>>> it > >>>>>>>>>> from > >>>>>>>>>>>> the > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML > >>>> tensor) > >>>>>>>> logical > >>>>>>>>>>> type. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Currently, the immediate need is to define a > >>>> specialized > >>>>>>> VECTOR > >>>>>>>>>> type > >>>>>>>>>>> so > >>>>>>>>>>>>>> that different engines and formats have a common > >>>> semantic > >>>>>>>>> contract > >>>>>>>>>>> for > >>>>>>>>>>>>>> interoperability. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If the main point of contention is just the > >>>> NaN/Inf/null > >>>>>>>>>> limitations, > >>>>>>>>>>>>> maybe > >>>>>>>>>>>>>> we can resolve this by adding a qualification to > >> the > >>>> name > >>>>>>> (like > >>>>>>>>>>> Andrew > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) > >>> to > >>>>>> avoid > >>>>>>>>>>> confusion > >>>>>>>>>>>>> with > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could > >>>> introduce a > >>>>>>>> boolean > >>>>>>>>>>> flag > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed > >>>>>> earlier. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52â¯AM Andrew Lamb < > >>>>>>>>>> [email protected]> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I am a little confused about the points of > >>>> disagreement > >>>>>>> here. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for > >> the > >>>>>>> specific > >>>>>>>>> use > >>>>>>>>>>> case > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used > >> for > >>>>>> vector > >>>>>>>>>>>> embeddings? > >>>>>>>>>>>>>> If > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the > >> name > >>>>>>>>>> (FINITE_VECTOR? > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It seems like there is precedent for defining > >>> logical > >>>>>> types > >>>>>>>>> that > >>>>>>>>>>>>> specify > >>>>>>>>>>>>>>> certain physical types and restrict their > >> contents > >>> to > >>>>>> help > >>>>>>>>>> readers > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON; > >>>>>> Defining > >>>>>>>> one > >>>>>>>>>> for > >>>>>>>>>>>>>> vector > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [1]: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34â¯PM Daniel Weeks < > >>>>>>>>> [email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits > >>>> their > >>>>>>>> usage, > >>>>>>>>>> but > >>>>>>>>>>>>> we're > >>>>>>>>>>>>>>>>> talking about a general-purpose file format > >>> that > >>>>> aims > >>>>>>> to > >>>>>>>> be > >>>>>>>>>>>> broadly > >>>>>>>>>>>>>>>>> applicable. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I agree that the format is intended to be > >> general > >>>>>>> purpose, > >>>>>>>>> but > >>>>>>>>>>>> types > >>>>>>>>>>>>>>> within > >>>>>>>>>>>>>>>> the format define specific representation and > >>>>> semantics > >>>>>>> to > >>>>>>>>>>> achieve > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> goal. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > >>>> it's > >>>>>>> quite > >>>>>>>>>>> trivial > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > >>> e.g. > >>>>>> NumPy > >>>>>>>> or > >>>>>>>>>>> Pandas > >>>>>>>>>>>>> (or > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > >>>>> computing > >>>>>>>>>> software). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The justification for restricting non-finite > >>>> values > >>>>> is > >>>>>>>>>> directly > >>>>>>>>>>>>>> related > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> the semantic meaning of a vector. Vectors have > >>>>>> specific > >>>>>>>>>>>>>> characteristics > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). > >>> Non-finite > >>>>>>> values > >>>>>>>>> are > >>>>>>>>>>>>> invalid > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> break those characteristics. I'd liken it to > >>>> storing > >>>>>>>>> non-utf8 > >>>>>>>>>>>> values > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>> string and still calling it a string. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed > >>>> length > >>>>>>> list, > >>>>>>>>>> which > >>>>>>>>>>>> is > >>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>> general and does not carry the same semantic > >>>> meaning. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I also don't think we need to bundle all > >>> similarly > >>>>>> shaped > >>>>>>>> use > >>>>>>>>>>> cases > >>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list). > >> Vector > >>>> is > >>>>> a > >>>>>>> well > >>>>>>>>>>>>> established > >>>>>>>>>>>>>>>> data type at this point and aligns with the > >>> general > >>>>>>> purpose > >>>>>>>>>>> nature > >>>>>>>>>>>> of > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> format (many other formats and databases > >> natively > >>>>>> support > >>>>>>>>>> this). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -Dan > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55â¯AM Antoine > >> Pitrou < > >>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a > >> écrit > >>> : > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the > >>>> clear > >>>>>>>>> use-case > >>>>>>>>>>> for > >>>>>>>>>>>>>>>> non-finite > >>>>>>>>>>>>>>>>>> values within a vector type. There has > >> been a > >>>> lot > >>>>>> of > >>>>>>>>>>>> theoretical > >>>>>>>>>>>>>>>>> discussion > >>>>>>>>>>>>>>>>>> here about use cases but what toolings > >>>> currently > >>>>>>>> produce > >>>>>>>>>>> these > >>>>>>>>>>>>>>> vectors > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> how do they function in downstream engines? > >>> I'd > >>>>>> like > >>>>>>> to > >>>>>>>>>> limit > >>>>>>>>>>>>>>>> introducing > >>>>>>>>>>>>>>>>>> types that have a wide range of values > >> which > >>>> are > >>>>>>> simply > >>>>>>>>> are > >>>>>>>>>>>>> failure > >>>>>>>>>>>>>>>> modes > >>>>>>>>>>>>>>>>>> of different flavors > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > >>>> it's > >>>>>>> quite > >>>>>>>>>>> trivial > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > >>> e.g. > >>>>>> NumPy > >>>>>>>> or > >>>>>>>>>>> Pandas > >>>>>>>>>>>>> (or > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > >>>>> computing > >>>>>>>>>> software). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector > >>> is > >>>> a > >>>>>> IETF > >>>>>>>>>>> standard > >>>>>>>>>>>> or > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8 > >> (where > >>>> the > >>>>>>>> values > >>>>>>>>>> are > >>>>>>>>>>>>>> expected > >>>>>>>>>>>>>>>>> to be valid according to the respective > >> specs) > >>> do > >>>>> not > >>>>>>>>> apply. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better > >> chance > >>>> of > >>>>>>>>> converging > >>>>>>>>>>> if > >>>>>>>>>>>> we > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data > >>> in > >>>>>>> Parquet > >>>>>>>>>> (with > >>>>>>>>>>> a > >>>>>>>>>>>>>>>>> potential limitation to non-nested types) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. representing the information that a given > >>>> column > >>>>>> (or > >>>>>>>>>> column > >>>>>>>>>>>>> chunk, > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ... both of which are useful on their own > >>> right, > >>>>> but > >>>>>> do > >>>>>>>> not > >>>>>>>>>>> have > >>>>>>>>>>>> to > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>> conflated and bundled together. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Regards > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Antoine. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
