Hi Everyone, Glad to see a Vector type being added. I have a few use cases I want to pursue once we have an agreed upon definition.
In regards to that definition discussion. I feel comfortable with the idea of the semantic meaning Dan raises though I can’t shake a slight discomfort (maybe FOMO) with limiting values in the physical definition. A few naive questions: 1. Would a physical definition with NANs & Inf+/- allow efficient encodings like ALP? 2. If the physical representation allows values that make encodings like ALP impossible, can a tighter Logical type allow different encodings? 3. Can we RLE or binpack a vector? Best Regards, Kurtis C. Wright > On Oct 14, 2026, at 12:45, Daniel Weeks <[email protected]> wrote: > > Hey Jan, > > There's a lot that I disagree with respect to the framing of these > arguments and how we should be thinking about introducing vector as a > logical type. There's a repeated tendency to let the physical > representation or performance drive discussion around types as opposed to > what it represents. > > In parquet, establishing semantic meaning is the purpose of logical types. > The logical type separates how to interpret a value from its physical > representation. We can represent a timestamp with INT64 or even a string > value, so why have a type for it? Even though a FIXED_SIZED_LIST and VECTOR > may have the exact same representation and performance, they have different > semantic meaning. Databases and engines across the ecosystem already > recognize this differentiation and have separate representations. The > choice to separate these isn't because the discussion is too tedious; > they're fundamentally different things. > > The finite requirement is what makes a vector a reflection of what it > represents. As we've discussed in the thread, non-finite values in the > representation violate the properties of a vector and the value ceases to > be a vector. On the other hand, a 0 vector is still a vector (you can > argue the utility of it, but it is a valid vector). Implementations can > choose what they validate or how to handle execution errors; that's an > implementation concern, not a requirement. Operations performed on a > vector depend on the context in which they are used, but the representation > doesn't need to solve scenarios where that application doesn't make sense. > > The guarantees or context needed to use a vector do not need to be inherent > in the type for it to be a useful type. We don't need the currency used > for decimal or whether a floating point is a normalized value as part of > the type to make use of those types. This conflates the use cases with the > type system and unnecessarily attempts to draw the context into the type > definition. Similarly, you don't need to capture normalization, model, > etc. as part of the type for it to be useful. > > So the reason we want to add a vector type (or FINITE_VECTOR) is because we > want to store vectors with the semantic meaning and requirements based on > what they represent. > > -Dan > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <[email protected]> wrote: >> >> A bit late to the party, but here we go: >> >> I have the strong sentiment that the FINITE_VECTOR type as it is currently >> being discussed will not be very useful and just increase implementation >> complexity. The arguments for the type are all quite weak (details see >> below). The FINITE part doesn't seem to yield any value and is not a useful >> restriction by itself. The vector (i.e. fixed length) part should rather be >> generalized instead of making a one-off thing. In general, the existing >> types are already working well for uncompressed vectors. The real challenge >> that would bring real benefits is supporting quantization. This is where >> you get orders of magnitude improvements to storage space and processing >> speed. >> >> Let's first discuss the FINITE part. So far, the only arguments I have read >> for this are: >> >> 1. many other vector databases have it >> 2. vectors are the hot stuff right now, so Parquet better boast having >> "native" type support for vectors >> 3. by focussing on this common special case, we can make rapid progress, >> while a general solution would take too long to specify >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the >> vector >> >> the first two arguments are quite weak by themselves. Vector databases are >> very specialized engines. Such engines can of course be very opinionated on >> their data types. A general format like Parquet must instead weigh whether >> a special type is worth the added complexity over a more general, widely >> applicable type. In this case, it is IMHO not at all. And just having a >> type called "vector" for advertisement reasons to make Parquet appear more >> modern seems a non-goal in the first place. >> >> The third "time to market" argument is valid for systems that don't need a >> stable interface. It is a very dangerous attitude for formats like Parquet >> that must be stable long term and should have a consolidated, concise, >> non-redundant type system. For such long term storage formats, we need to >> take the time to build the right thing, as any shortcuts we take will stay >> as blemishes in the format forever. If I read the discussion correctly, >> this is actually the main argument as of now: We have found that it is hard >> to align on how a fixed size list type should look like, so we want to go >> the easier route of a specialized type. This seems like a >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would we >> want this, if Parquet already had a fixed size list type? >> >> So finally, the only technical argument is the fourth one: "It signals >> semantics and ensures that no bad data lands in the format". I see two big >> problems with this: >> >> 1. Readers *still* need to test for NaN / INF because Parquet files may be >> corrupt. Nothing can stop a malicious or buggy writer from physically >> writing NaN or INF. *So you save no single line of code in a reader. You >> save no branches, as you still need to handle the INF/NaN case.* So this >> doesn't help with performance at all. The only difference is, that the >> error message changes from "found an invalid INF value in the data" to >> "Your parquet file seems to be corrupt". It is actually quite the opposite, >> readers and writers get more complex having to handle this new type. >> >> If you now argue that it's not about performance, it's about defined >> semantics and erroring out early in writers, then the second problem >> arises: >> >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple >> guarantees that you need for sensible vector processing. If "no >> INF/NaN/NULL" was the only guarantee you needed, I would agree that >> FINITE_VECTOR is useful, but it is by no means the only guarantee you need, >> instead, depending on the distance metric, you also need the following >> guarantees. E.g.: >> >> - No vector may be the 0 vector, otherwise, e.g., cosine similarity is >> undefined. Cosine similarity is often used as default, so this is >> actually >> quite important in practice. >> - If we want to use a dot product, then all vectors need to guarantee a >> length of 1. This would actually be quite useful for performance >> reasons. >> - The vectors must come from the same model as the vector used for the >> search. Otherwise, the search yields a numeric result, but that result >> has >> no sensible meaning. >> >> So, what use is a type that just guarantees part of the things you need for >> vector search but not others? You still need additional guarantees in >> addition to that type. What makes INF/NaN so important in comparison to, >> say, "no 0 vector present", that it warrants an own type, while the other >> guarantees do not? If we really want this type as a basis for cross-engine >> sharing of vectors, then it needs to give *all* guarantees. >> >> I'm not against a specialized vector type if it really carries *all* >> information to be useful by itself. If it carried all that information, >> then a reading engine could indeed validate that only meaningful vector >> searches are executed. But as long as it is just a half-baked thing, it is >> not useful by itself and just a burden on each implementer and maintainer. >> If we need extra information anyway, why not just stick to a general type >> like VECTOR (without FINITE). >> >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what >> advantage you will *really* get over just a (potentially fixed size) List >> of Float? As discussed, it's not performance. It's also not a semantic >> marker that is sufficient by itself. It's just a small hint that allows >> writers to error out once they see a NaN or INF in the input, while they >> will still happily write a bogus 0 vector. This seems to be of zero >> benefit. >> >> Anecdotally, we currently handle petabytes of vectors in production in >> Data360. We simply store them in a List of Float, with an optimized R-Level >> reader implementation that rapidly checks that all vectors are the same >> length. This solution works just fine. The overhead of R-level decoding is >> barely measurable. The check is super fast and R-levels are tiny in >> comparison to the payload vectors, so also the added I/O is negligible. >> This is also why I'm so firm in my sentiment: We are running a production >> grade system processing trillions of vectors right now and see no big >> issues with the types that Parquet currently provides. >> >> As I don't just want to be a naysayer and point out problems with the >> current design, here are the directions that I would propose indeed: >> >> - a general fixed size list type. Don't do shortcuts just because the >> discussion is too tedious. >> - a vector type that indeed carries enough information to be really >> useful for vector search by itself. (e.g., model name / version, >> optional >> guarantees for no 0 vector / normalized vector length). With such a >> type, >> you could really do well defined vector searches even if the writing >> engine >> is completely different to the reading one. >> - Standardize quantizations. This will be a real game changer for >> performance. >> >> Cheers, >> Jan >> >> >> >> >> >> >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <[email protected]>: >>> >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a >> good >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural. >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first? >>> >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < >>> [email protected]> wrote: >>> >>>> I suggest we go with VECTOR for the finite vector to keep the name the >>> same >>>> as other engines/type systems. If we want a non-finite one in the >> future >>> we >>>> can name that something else or we add fixed size list that will serve >>> the >>>> same purpose without a special name. >>>> >>>> >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> wrote: >>>> >>>>> A logical type would: >>>>> - Describe data for users and readers to understand what the file >>>> contains >>>>> - Signal that repetition and definition levels will reflect a fixed >>>> number >>>>> of elements, which implementations can use to optimize decoding >>>>> - Guarantee that there are no null elements (because the type cannot >> be >>>>> applied to a list with optional elements) >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity >> values >>>> so >>>>> that the vectors have a finite length and are suitable for distance >>>>> calculations >>>>> >>>>> The finite value restriction is a restriction on writers to prevent >> bad >>>>> data from being written. >>>>> >>>>> Ignoring logical types that are not understood is a rule for >> readers. A >>>>> reader can handle vectors and pass it on to another layer. For >>> instance, >>>> a >>>>> Parquet reader that doesn't support geometry passes values as >> `binary`. >>>>> That binary value can be passed to a geometry UDF, even if the >>> underlying >>>>> engine and Parquet reader doesn't have support. >>>>> >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a >> list >>> of >>>>> floats along to a UDF. But it is still important to the user that bad >>>>> values are not present in the data. In your example, bad values are >>>> vectors >>>>> that cause the centroid calculation to fail, like vectors with >> infinite >>>>> length. >>>>> >>>>> Ryan >>>>> >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < >>>>> [email protected]> >>>>> wrote: >>>>> >>>>>> Could we clarify the use cases for a VECTOR annotation on >>> LIST<FLOAT>? >>>>>> Specifically, what does it make possible that's impossible today, >> and >>>>> what >>>>>> does it make easier that's currently hard? >>>>>> >>>>>> I'm struggling to find a strong answer. I'd like every engine to >>> offer >>>>>> vector functions on lists of floats without needing a new logical >>> type. >>>>>> >>>>>> To be clear, I'm still enthusiastic about a vector type that >>> describes >>>>> how >>>>>> to unpack and scale composite representations (quantised codes plus >>>>> scales, >>>>>> codebooks, etc.), in the direction ML is going. That adds >> information >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already >>>>>> understand the physical type, and many already offer vector >> functions >>>> on >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are >>>> cheap >>>>>> enough that engines should do them anyway. The label also doesn't >> say >>>>> what >>>>>> you can do with the column. Knowing that a column holds embeddings >>>>> doesn't >>>>>> say how they can be used, just as zipcode and age are both integers >>> but >>>>>> multiplying them is meaningless. >>>>>> >>>>>> My concern is the opposite effect. We're establishing that engines >>>> should >>>>>> ignore logical types they don't understand. So a tool that doesn't >>> know >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute >>>>> centroids >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines >>> restrict >>>>>> embedding functions to VECTOR-annotated columns, that file stops >>>> working >>>>>> with them, through no fault of the user. An annotation that doesn't >>>>> change >>>>>> what the data is shouldn't be able to make it unusable. >>>>>> >>>>>> Am I missing a use case? >>>>>> >>>>>> Best, Will >>>>>> >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> >> wrote: >>>>>> >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many >>> use >>>>>> cases. >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and >> not >>>> to >>>>>> fit >>>>>>> every use cases into the FIXED_LENGTH_LIST. >>>>>>> >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> >> wrote: >>>>>>> >>>>>>>> Going through this thread again, it looks like there are a few >>>> things >>>>>>> that >>>>>>>> people generally agree on: >>>>>>>> >>>>>>>> - In the short term, we want to introduce a logical type or >>> types; >>>> a >>>>>>> better >>>>>>>> physical representation can be done later. We can focus on >>> logical >>>>>> types. >>>>>>>> - There is precedent in logical types for restricting values, >>> like >>>>> UTF8 >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree >>>> that >>>>> we >>>>>>>> could restrict vector elements to non-nested or primitive >> types. >>>>>>>> - There are many examples of databases that restrict their >> vector >>>>> types >>>>>>> to >>>>>>>> non-null, finite elements. >>>>>>>> - Some people want to have write-time guarantees that vectors >>> have >>>>> only >>>>>>>> non-null, finite elements >>>>>>>> - Some people want to extend this work to include tensors, >>> shorter >>>>>>> floating >>>>>>>> point representations, variable lengths, NaN or Infinity >>> elements, >>>>> etc. >>>>>>>> >>>>>>>> It sounds like we are nearly there, but have two halves of the >>>>>> community >>>>>>>> that care about different things. That just means we should >> have >>>>>> multiple >>>>>>>> types. It's entirely reasonable to want a type that allows NaN >>>>> values, >>>>>>> and >>>>>>>> it's also reasonable for those of us building databases, table >>>>>> standards, >>>>>>>> and vector indexes to care about guarantees that elements are >>>> finite. >>>>>>>> There's more than enough examples to show that this is a >>> legitimate >>>>> use >>>>>>>> case. >>>>>>>> >>>>>>>> Since I don't see the concerns of the different use cases >>> changing, >>>>> it >>>>>>>> looks like we need to find a compromise to move forward. I >>> propose >>>>> that >>>>>>> we >>>>>>>> introduce a FINITE_VECTOR type and separately a >> FIXED_LENGTH_LIST >>>> or >>>>>>>> another VECTOR type that allows infinite and NaN values. >>>>>>>> >>>>>>>> Can we agree on a compromise? Is there a strong reason to try >> to >>>> fit >>>>>>> these >>>>>>>> use cases into a single type? What if that means not making >>>> progress? >>>>>>>> >>>>>>>> Ryan >>>>>>>> >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thinking about this more, I am also fine with pulling out the >>>>>> “NaN/Inf” >>>>>>>>> requirement. Either into a type boolean (my preference), or >>>> putting >>>>>> no >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754 >> float >>>>>> numbers. >>>>>>>> If >>>>>>>>> some top-level application wants to put additional >> constraints >>> on >>>>> the >>>>>>>> data >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do >>> so >>>>> one >>>>>>>> layer >>>>>>>>> above. It seems like this was the biggest point of discussion >>>>>>> remaining. >>>>>>>>> All these options work for me, and we should move forward >> with >>>> this >>>>>>>>> proposal. >>>>>>>>> >>>>>>>>> Concerning additional element types like bf16, they can be >>>>> introduced >>>>>>> as >>>>>>>>> Parquet types (at least in the current LIST-based spec >>> proposal). >>>>>>>>> >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a >>>>> variable >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I >> think >>>>> that >>>>>> a >>>>>>>>> dedicated logical VECTOR type still provides enough benefits >> by >>>>>>>> describing >>>>>>>>> the semantics of the data, and allowing to capture future >>> changes >>>>>>> around >>>>>>>>> statistics. In that case, the length is better captured as an >>>> exact >>>>>>>> length >>>>>>>>> for this type specifically. I’m also not sure about other use >>>> cases >>>>>> for >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just >>> for >>>>> the >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough >>> value >>>> in >>>>>> my >>>>>>>>> view. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Philipp >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < >>>>>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond >> normalized >>>>>>> embeddings >>>>>>>>> and >>>>>>>>>> engines really aren't going to be able to elide all the >>> checks >>>>>>> anyway. >>>>>>>>>> >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :) >>>>>>>>>> >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't >>>> express >>>>>> this >>>>>>>> in >>>>>>>>>> Parquet. Engines can write opaque blobs they understand in >>> the >>>>>>>> metadata >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but >>> this >>>>>> isn't >>>>>>>>>> standardized, portable, nor interoperable. >>>>>>>>>> >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift >>>> fields >>>>>> (my >>>>>>>>> pref) >>>>>>>>>> can generically support VECTOR(N) whilst also being more >>>> broadly >>>>>>> useful >>>>>>>>> for >>>>>>>>>> other use cases and data types? >>>>>>>>>> >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and >>>>>>> MAX_LENGTH >>>>>>>>> for >>>>>>>>>> Parquet's variable-length physcial types. These would >> count >>>>>> logical >>>>>>>>> units >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes. >>>>>>>>>> >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as >>>> LIST(N) >>>>>> with >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even >> different >>>>>>> engines - >>>>>>>>> can >>>>>>>>>> pick up these constraints if they support doing so. We >> don't >>>>>>> actually >>>>>>>>> need >>>>>>>>>> a logical type to express a VECTOR then. >>>>>>>>>> >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to >> be >>>>>>> expressed >>>>>>>> on >>>>>>>>>> UTF8 columns etc. >>>>>>>>>> >>>>>>>>>> This also seems generally useful in the future as we >> approach >>>>>>>> expressing >>>>>>>>>> other tensor data shapes and types. >>>>>>>>>> >>>>>>>>>> This won't change the mechanics: the storage type doesn't >>>> enforce >>>>>>> these >>>>>>>>>> constraints and engines must check them. It's previously >>>>>> established >>>>>>>> that >>>>>>>>>> performance is just a software problem and there is no >>>>> fundamental >>>>>>>>>> performance advantage to the fixed size types in >> Dremel-like >>>>>> formats >>>>>>>> like >>>>>>>>>> Parquet. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Will >>>>>>>>>> >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < >>>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> After listening in on the last meeting Wednesday, I have >>> been >>>>>>>> thinking >>>>>>>>>>> about this quite a bit, and thought I would share my 2 >>> cents. >>>>>>>>>>> >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as >>>>> proposed >>>>>>> in >>>>>>>>> the >>>>>>>>>> PR >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>. >>>>>>>>>>> >>>>>>>>>>> While Divjot and others note the specialization for >> UTF-8, >>>>>> Parquet >>>>>>>>>> already >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings >>>>>>>>>>> < >>>>>>>> >>>> https://github.com/apache/parquet-format/blob/master/README.md#types >>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new >>>>>>> restriction >>>>>>>>> on >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE >>>>> columns >>>>>>>> have >>>>>>>>>>> never had such a rule. >>>>>>>>>>> >>>>>>>>>>> It would also not match how the systems that actually >>> produce >>>>>> these >>>>>>>>>> vectors >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark, >>>> DuckDB, >>>>>>>> NumPy, >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and >>>> OpenBLAS. >>>>>>>>>>> >>>>>>>>>>> pgvector's "all elements must be finite" rule is an >> insert >>>>>> policy. >>>>>>>>> While >>>>>>>>>> I >>>>>>>>>>> think Parquet as a general-purpose format should allow >> them >>>> out >>>>>> of >>>>>>>> the >>>>>>>>>> box. >>>>>>>>>>> It is a user system decision to disallow writing these >>> values >>>>> or >>>>>> to >>>>>>>>> throw >>>>>>>>>>> errors when encountering them. >>>>>>>>>>> >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector >>>> math. >>>>>>>> Finite >>>>>>>>>>> coordinates still overflow. For instance: magnitude = >>>>>> ||[FLOAT_MAX, >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows >> to >>>>> +Inf. >>>>>>>>> PyTorch >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm() >> is >>>> inf >>>>>> in >>>>>>>>> fp32. >>>>>>>>>>> >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while >>> the >>>>>>> VECTOR >>>>>>>>>> itself >>>>>>>>>>> was required to be finite. The ban only forbids storing >>>>> specials >>>>>> in >>>>>>>> the >>>>>>>>>>> coordinates, and does not prevent even simple downstream >>>>>>> computations >>>>>>>>>> from >>>>>>>>>>> producing them. >>>>>>>>>>> >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional >>> bool >>>>>>>>>>> VectorType.finite (default false). Engines that want >>>>>> pgvector-like >>>>>>>>>>> rejection can require the flag = true. Engines that use >>>> NumPy / >>>>>>>>> PyTorch / >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the >>> same >>>>>>>> reason, I >>>>>>>>>> do >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Sebastian >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < >>>>>>>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I agree with moving forward with the proposal and >>> renaming >>>>>> VECTOR >>>>>>>> to >>>>>>>>>>>> FINITE_VECTOR or something >>>>>>>>>>>> similar if others feel that better communicates its >>>> intent. I >>>>>> see >>>>>>>>> value >>>>>>>>>>> in >>>>>>>>>>>> having a vector type that guarantees >>>>>>>>>>>> its values are mathematically-correct vectors for a few >>>>>> reasons: >>>>>>>>>>>> >>>>>>>>>>>> 1. As others have mentioned, this is similar to the >>> STRING >>>>>>> logical >>>>>>>>> type >>>>>>>>>>>> mandating UTF-8. >>>>>>>>>>>> 2. It is not possible to rely on statistics for >> detecting >>>> NaN >>>>>> or >>>>>>>>> +-inf >>>>>>>>>>>> values as all statistics are optional. >>>>>>>>>>>> To my knowledge, there is no precedent for requiring >> that >>>>>> writers >>>>>>>>> emit >>>>>>>>>>>> stats for specific logical types. >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow >> through >>>> the >>>>>>> engine >>>>>>>>> and >>>>>>>>>>>> block them for specific functionality. >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't >> think >>>>> this >>>>>> is >>>>>>>> as >>>>>>>>>> easy >>>>>>>>>>>> as it sounds and Lance has run >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples). >>> Their >>>>>>> Python >>>>>>>>> [3] >>>>>>>>>>> and >>>>>>>>>>>> Rust [4] APIs default to erroring on >>>>>>>>>>>> vectors that contain NaN values. >>>>>>>>>>>> >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I >>>> don't >>>>>>> think >>>>>>>>>> this >>>>>>>>>>>> implies the lack of domain-specific types. >>>>>>>>>>>> Instead, this broadens the use cases where Parquet >> would >>> be >>>>>>>>> considered >>>>>>>>>> a >>>>>>>>>>>> good fit. I also fully agree with >>>>>>>>>>>> continuing work on a general fixed-size list type if we >>>>> believe >>>>>>>> there >>>>>>>>>> are >>>>>>>>>>>> non-vector use cases that would benefit >>>>>>>>>>>> from such guarantees at the storage layer. >>>>>>>>>>>> >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648 >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607 >>> (see >>>>>>> point 2 >>>>>>>>> in >>>>>>>>>>>> "root >>>>>>>>>>>> cause") >>>>>>>>>>>> [3] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 >>>>>>>>>>>> [4] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via >>> dev < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I also favor moving forward with the proposal as is. >> I >>>> see >>>>>>> strong >>>>>>>>>> value >>>>>>>>>>>> and >>>>>>>>>>>>> use cases for the numeric vector type, and I agree >> with >>>> the >>>>>>>>>>> restrictions >>>>>>>>>>>>> around NaN/Inf values and element types in general. >>>>> Precedent >>>>>>>>> exists >>>>>>>>>> in >>>>>>>>>>>>> other types that restrict which physical type values >>> are >>>>>>>> logically >>>>>>>>>>> valid >>>>>>>>>>>>> (e.g., the geospatial types needing to be >> WKB-encoded). >>>> A >>>>>>>> general >>>>>>>>>>>>> fixed-size list also has value, but I think it makes >>>> sense >>>>> to >>>>>>>>>> separate >>>>>>>>>>>> the >>>>>>>>>>>>> two types. The semantics are relevant, and I can see >>> how >>>>> the >>>>>>> two >>>>>>>>>> types >>>>>>>>>>>>> might diverge further in the future regarding >>> statistics >>>> or >>>>>>>>>> encodings. >>>>>>>>>>>>> >>>>>>>>>>>>> I would be okay with renaming the vector type, but it >>>> seems >>>>>>> like >>>>>>>>> the >>>>>>>>>>>> VECTOR >>>>>>>>>>>>> name is already used across other engines mentioned >> in >>>> this >>>>>>>> thread. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Philipp >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < >>>> [email protected]> >>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should >>> treat >>>>> the >>>>>>>>> general >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout, >> and >>>>>> separate >>>>>>>> it >>>>>>>>>> from >>>>>>>>>>>> the >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML >>>> tensor) >>>>>>>> logical >>>>>>>>>>> type. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Currently, the immediate need is to define a >>>> specialized >>>>>>> VECTOR >>>>>>>>>> type >>>>>>>>>>> so >>>>>>>>>>>>>> that different engines and formats have a common >>>> semantic >>>>>>>>> contract >>>>>>>>>>> for >>>>>>>>>>>>>> interoperability. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If the main point of contention is just the >>>> NaN/Inf/null >>>>>>>>>> limitations, >>>>>>>>>>>>> maybe >>>>>>>>>>>>>> we can resolve this by adding a qualification to >> the >>>> name >>>>>>> (like >>>>>>>>>>> Andrew >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) >>> to >>>>>> avoid >>>>>>>>>>> confusion >>>>>>>>>>>>> with >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could >>>> introduce a >>>>>>>> boolean >>>>>>>>>>> flag >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed >>>>>> earlier. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < >>>>>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am a little confused about the points of >>>> disagreement >>>>>>> here. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for >> the >>>>>>> specific >>>>>>>>> use >>>>>>>>>>> case >>>>>>>>>>>>> of >>>>>>>>>>>>>>> fixed length arrays with non-finite values used >> for >>>>>> vector >>>>>>>>>>>> embeddings? >>>>>>>>>>>>>> If >>>>>>>>>>>>>>> so, maybe we could add a qualification to the >> name >>>>>>>>>> (FINITE_VECTOR? >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems like there is precedent for defining >>> logical >>>>>> types >>>>>>>>> that >>>>>>>>>>>>> specify >>>>>>>>>>>>>>> certain physical types and restrict their >> contents >>> to >>>>>> help >>>>>>>>>> readers >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON; >>>>>> Defining >>>>>>>> one >>>>>>>>>> for >>>>>>>>>>>>>> vector >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < >>>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits >>>> their >>>>>>>> usage, >>>>>>>>>> but >>>>>>>>>>>>> we're >>>>>>>>>>>>>>>>> talking about a general-purpose file format >>> that >>>>> aims >>>>>>> to >>>>>>>> be >>>>>>>>>>>> broadly >>>>>>>>>>>>>>>>> applicable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I agree that the format is intended to be >> general >>>>>>> purpose, >>>>>>>>> but >>>>>>>>>>>> types >>>>>>>>>>>>>>> within >>>>>>>>>>>>>>>> the format define specific representation and >>>>> semantics >>>>>>> to >>>>>>>>>>> achieve >>>>>>>>>>>>> that >>>>>>>>>>>>>>>> goal. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and >>>> it's >>>>>>> quite >>>>>>>>>>> trivial >>>>>>>>>>>> to >>>>>>>>>>>>>>>>> produce vectors with non-finite values using >>> e.g. >>>>>> NumPy >>>>>>>> or >>>>>>>>>>> Pandas >>>>>>>>>>>>> (or >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific >>>>> computing >>>>>>>>>> software). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The justification for restricting non-finite >>>> values >>>>> is >>>>>>>>>> directly >>>>>>>>>>>>>> related >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> the semantic meaning of a vector. Vectors have >>>>>> specific >>>>>>>>>>>>>> characteristics >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). >>> Non-finite >>>>>>> values >>>>>>>>> are >>>>>>>>>>>>> invalid >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> break those characteristics. I'd liken it to >>>> storing >>>>>>>>> non-utf8 >>>>>>>>>>>> values >>>>>>>>>>>>>> in >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> string and still calling it a string. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed >>>> length >>>>>>> list, >>>>>>>>>> which >>>>>>>>>>>> is >>>>>>>>>>>>>> more >>>>>>>>>>>>>>>> general and does not carry the same semantic >>>> meaning. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also don't think we need to bundle all >>> similarly >>>>>> shaped >>>>>>>> use >>>>>>>>>>> cases >>>>>>>>>>>>>> into >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list). >> Vector >>>> is >>>>> a >>>>>>> well >>>>>>>>>>>>> established >>>>>>>>>>>>>>>> data type at this point and aligns with the >>> general >>>>>>> purpose >>>>>>>>>>> nature >>>>>>>>>>>> of >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> format (many other formats and databases >> natively >>>>>> support >>>>>>>>>> this). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Dan >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine >> Pitrou < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a >> écrit >>> : >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the >>>> clear >>>>>>>>> use-case >>>>>>>>>>> for >>>>>>>>>>>>>>>> non-finite >>>>>>>>>>>>>>>>>> values within a vector type. There has >> been a >>>> lot >>>>>> of >>>>>>>>>>>> theoretical >>>>>>>>>>>>>>>>> discussion >>>>>>>>>>>>>>>>>> here about use cases but what toolings >>>> currently >>>>>>>> produce >>>>>>>>>>> these >>>>>>>>>>>>>>> vectors >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> how do they function in downstream engines? >>> I'd >>>>>> like >>>>>>> to >>>>>>>>>> limit >>>>>>>>>>>>>>>> introducing >>>>>>>>>>>>>>>>>> types that have a wide range of values >> which >>>> are >>>>>>> simply >>>>>>>>> are >>>>>>>>>>>>> failure >>>>>>>>>>>>>>>> modes >>>>>>>>>>>>>>>>>> of different flavors >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and >>>> it's >>>>>>> quite >>>>>>>>>>> trivial >>>>>>>>>>>> to >>>>>>>>>>>>>>>>> produce vectors with non-finite values using >>> e.g. >>>>>> NumPy >>>>>>>> or >>>>>>>>>>> Pandas >>>>>>>>>>>>> (or >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific >>>>> computing >>>>>>>>>> software). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It's not like the general concept of a vector >>> is >>>> a >>>>>> IETF >>>>>>>>>>> standard >>>>>>>>>>>> or >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8 >> (where >>>> the >>>>>>>> values >>>>>>>>>> are >>>>>>>>>>>>>> expected >>>>>>>>>>>>>>>>> to be valid according to the respective >> specs) >>> do >>>>> not >>>>>>>>> apply. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This discussion would IMHO have a better >> chance >>>> of >>>>>>>>> converging >>>>>>>>>>> if >>>>>>>>>>>> we >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data >>> in >>>>>>> Parquet >>>>>>>>>> (with >>>>>>>>>>> a >>>>>>>>>>>>>>>>> potential limitation to non-nested types) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. representing the information that a given >>>> column >>>>>> (or >>>>>>>>>> column >>>>>>>>>>>>> chunk, >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ... both of which are useful on their own >>> right, >>>>> but >>>>>> do >>>>>>>> not >>>>>>>>>>> have >>>>>>>>>>>> to >>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>> conflated and bundled together. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Antoine. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
