lol and so you see I asked an llm to proof read my reply and pasted hastily :D But I did write my reply. You can even reverse-engineer my human version from the edits :D
:embarrassed: Will On Mon, 19 Oct 2026 at 12:11, Will Edwards <[email protected]> wrote: > It reads well. Three small fixes: > > 1. **"back in May or so"** is ambiguous. The fixed-size-list-for-tensors > framing is from Rok's opening post in May 2024. The May 2026 design doc was > already mostly about performance and layout. "Originally" avoids anyone > correcting the date. > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite > right. The NaN-for-missing behaviour comes from `VectorAssembler` and > `Imputer` and applies to Spark ML vectors generally, dense or sparse. > "Spark's feature vectors, dense and sparse," fixes it. > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and > DuckDB and pretty much…" has a doubled "and". > > The last sentence of the invariants paragraph could be clearer. "Although > that doesn't mean readers shouldn't check on read" has a double negative > and is easy to misread. "Though careful readers will still check on read" > is simpler. > > With those applied: > > --- > > Hi all, > > The scope of this discussion keeps moving :). IIRC it originally was about > fixed-size lists for ML tensors, became a performance question, and is now > about whether vectors are mathematically valid. Each step made sense, but I > don't think we have consensus on where we've ended up? > > I think there is much useful work in what we left behind. Many ML vectors > aren't embeddings for search. E.g. Spark's feature vectors, dense and > sparse, routinely use NaN to mark missing values, and quantised formats > pack codes with scales in ways Parquet can't describe today. ML-oriented > formats like Lance are moving towards tensors, quantisation and indexes. A > type that told engines how to interpret that kind of data would enable new > things. An annotation on LIST<FLOAT> doesn't. > > My big concern with the current focus on embeddings needing a logical type > is compatibility. Today LIST<FLOAT> is the common format for embeddings, > and Spark, BigQuery, DuckDB and pretty much every other library I can think > of offer distance functions on plain lists, so any engine can use any file. > If engines start requiring a VECTOR annotation, existing files and anything > rewritten by a tool that doesn't know the type stop working. If they don't > require it, the annotation adds nothing. We've seen this before with > strings written without the UTF8 annotation, which engines still have to > cope with years later. > > On invariants, I agree some users may want the writer to check what they > write as they write it, but that can be a writer option rather than a type. > Readers that want evidence could get it from extra statistics or metadata, > though careful readers will still check on read. > > If a finite fixed-length vector contract is useful to table formats, > perhaps Iceberg is the place for it, with Parquet storing plain lists. I'd > raise the same worry there, though: if tables start rejecting untagged > columns, we've just moved the split up a layer. > > Best, > Will > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <[email protected]> wrote: > >> I agree with much of what Dan said, but I'll add some to this thread as >> well. >> >> First, I want to address the arguments that were pointed out: >> 1. Other vector databases have finiteness requirements >> 2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon >> 3. A general solution will take too long, finiteness will make faster >> progress >> 4. To give a guarantee to readers that no INF/NaN/NULL will be in the >> vector >> >> I don't think these represent our actual motivation or position, which >> makes them function as a straw-man. This is probably not what was >> intended, but I want to make sure that the misunderstanding >> doesn't continue. For one thing, we aren't saying that we need finite >> vectors because they are "hot stuff". >> >> We want to build tools that use them on top of Parquet. I think it should >> also be clear that we aren't advocating for finite elements blindly just >> because other engines do (cargo cult design!). There are real >> interoperability issues if we choose to go a different direction so this >> is >> a "Chesterton's Fence" situation. It's important to understand why >> those databases >> made that choice. I've also pointed this out as confirmation that we're >> going down a reasonable and established path. >> >> The third argument sounds like a simple misunderstanding. We're pursuing a >> logical type right now because that's forward compatible and we want to >> label the data properly and set expectations for the type. Physical type >> changes break forward compatibility and need to be released in a new >> format >> version . I think this is the source of the confusion since it has been in >> the same discussion, but the choice about finiteness is separate. >> >> That brings us to the last argument, whether there is value in writers >> participating in guarantees. I think there's value in adding requirements >> for writers. It's up to readers how they want to use those requirements, >> but a writer requirement allows a policy of "garbage in garbage out" >> rather >> than being forced to handle bad data. Why would a reader need to run >> validation on every element just in case a writer had a bug? And why >> should >> the potential for writer bugs mean that it has to be the reader's problem? >> If a writer produces bad data, we can absolutely reject it and tell the >> writer to fix the bug. >> >> I'm also puzzled by the assertion that writer guarantees have value, but >> only if there are other, more specific guarantees in addition. This seems >> to cede the point that a contract between readers and writers makes sense, >> but only if it is a _specific_ contract. But the proposed additions don't >> make sense in the same way. >> >> Take, for example, the suggestion to also disallow zero-length vectors. At >> a glance this seems similar, but I think it assumes a specific context and >> is better suited to be a constraint for some use cases rather than part of >> what is considered valid for the type. That's because zero-length vectors >> are valid vectors with a length. You can calculate Euclidean or Manhattan >> distance and use them in many contexts. They don't work if you're >> normalizing to a length of 1, but it's a choice to normalize and use a dot >> product to calculate cosine distance. And at this point, we're talking >> about the usefulness of restrictions for the type, not a Platonic ideal. >> >> I think the restriction that makes the most sense is that vectors have a >> length and can be compared in distance calculations. 0-length vectors are >> valid because they point somewhere and have a length. We shouldn't >> restrict >> uses to only cosine distance over normalized vectors. Those are use case >> dependent choices and there are other valid ones. I also don't think that >> the model used to produce a vector is part of the vector type, just like >> the method used to produce a metric doesn't define its type: cosine >> similarity produces values between -1 and 1, but we store it in a `float`, >> not a `float(cosine)`. >> >> Why does it make sense to have a finiteness restriction as part of the >> type? Because that's what a (mathematical) vector is. A simple search >> gives >> a pretty common definition: "A vector is an object that has both magnitude >> and a direction". If a dimension/element is NaN, null, or infinite, the >> vector doesn't point anywhere and it doesn't have a magnitude. Not only >> can >> it not be used, it doesn't meet the most basic definition of the type. >> >> Ryan >> >> On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <[email protected]> wrote: >> >> > > >> > > >> > > >> > Daniel, >> > >> > alright, so no physical optimizations intended. We only want this for >> > semantic reasons, not for any significant physical optimization. If >> we're >> > all on the same page here, that is good 👍. >> > >> > To phrase my remaining problem with this more concretely: >> > *Let's say we add this type. What cool new stuff can we now do that we >> > couldn't do before?* >> > That is the litmus test that a type has to pass to be useful IMHO. And >> here >> > I can't come up with a tangible use case that wasn't possible before. >> > Especially if we one day have fixed size lists in Parquet, the remaining >> > niche for this type becomes so small, that the added complexity isn't >> worth >> > the benefit IMHO. >> > >> > As I guess most people on here are active maintainers of a Parquet >> > implementation: Concretely, what would you be using this type for if we >> had >> > it? Would you allow math only on fields tagged with this type? What if >> that >> > math yields INF or NaN? Not allowed? >> > >> > Especially if we also discuss having both VECTOR and FINITE_VECTOR. How >> > will you guys wire this up with the type systems of your engines? Do >> your >> > engines also have two types here? Do you type-decay a FINITE_VECTOR to a >> > VECTOR once someone does a multiplication that could yield INF? By >> itself, >> > a FINITE_VECTOR sounds good. Holistically looked at from a type system >> > perspective, it comes with a lot of complexity. Do the benefits really >> > outweigh the complexity here? That's why I would rather have the bare >> > minimum: A fixed size array (list) of floats or at least just a vector >> > without finite. Then all these detailed questions go away. >> > >> > I'll take this one step further... Why is there any debate about this at >> > > all? Why can't a type simply be added to the *specification* as an >> > > administrative detail and be done? >> > >> > >> > Andrew, >> > >> > Parquet is supposed to be a lingua franca of data. The specification is >> the >> > authoritative truth for this language and all implementations are >> expected >> > to implement as much as possible from this spec, so anything in the spec >> > does causes implementation effort. So just adding anything to this spec >> > without debate is not a good idea. It is even the other way around: We >> need >> > to have way higher rigor with the spec than we need to have with any >> > implementations. Implementations can be changed, the spec needs to be >> > stable. >> > >> > A tag-only type that doesn't come with any restrictions can be added >> with >> > little effort. But here we want to tie restrictions to it (no NaN/Inf), >> so >> > we would expect implementations to enforce these. So there is some >> > implementation effort expected here. >> > >> > Also, even if we only wanted a tag type, I would still argue that you >> > shouldn't add those without a debate. What if A wants XML, B wants >> YAML, C >> > wants quaternions, D wants address, E wants phone number. If we just >> added >> > any types that are "just tags", Parquet logical types would just become >> a >> > trash dump of any type that anyone ever thought to be useful. So >> rigorous >> > assessment of each new type candidate is necessary. >> > >> > Finally, many engines couple parquet with another type system (e.g., >> Arrow >> > or SQL). So even with a tag only type, you at least still have the >> > implementation burden of wiring this up with your engine's type system. >> > >> > So, just because logical types are "just" logical, that doesn't mean we >> can >> > add them without any rigor. >> > >> > >> > A few naive questions: >> > > 1. Would a physical definition with NANs & Inf+/- allow efficient >> > > encodings like ALP? >> > > 2. If the physical representation allows values that make encodings >> like >> > > ALP impossible, can a tighter Logical type allow different encodings? >> > > 3. Can we RLE or binpack a vector? >> > >> > >> > The simple general answer here is that all these things do not require a >> > logical type. Writers can already do any such optimization if the actual >> > data contains no NaNs & Infs. All these physical optimizations just >> require >> > no actual NaNs/Infs to be present in a page, they don't need a guarantee >> > that the whole file won't contain any. >> > >> > Cheers, >> > Jan >> > >> > >> > >> > >> > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <[email protected]>: >> > >> > > Hi Everyone, >> > > >> > > Glad to see a Vector type being added. I have a few use cases I want >> to >> > > pursue once we have an agreed upon definition. >> > > >> > > In regards to that definition discussion. I feel comfortable with the >> > idea >> > > of the semantic meaning Dan raises though I can’t shake a slight >> > discomfort >> > > (maybe FOMO) with limiting values in the physical definition. >> > > >> > > A few naive questions: >> > > 1. Would a physical definition with NANs & Inf+/- allow efficient >> > > encodings like ALP? >> > > 2. If the physical representation allows values that make encodings >> like >> > > ALP impossible, can a tighter Logical type allow different encodings? >> > > 3. Can we RLE or binpack a vector? >> > > >> > > Best Regards, >> > > Kurtis C. Wright >> > > > On Oct 14, 2026, at 12:45, Daniel Weeks <[email protected]> wrote: >> > > > >> > > > Hey Jan, >> > > > >> > > > There's a lot that I disagree with respect to the framing of these >> > > > arguments and how we should be thinking about introducing vector as >> a >> > > > logical type. There's a repeated tendency to let the physical >> > > > representation or performance drive discussion around types as >> opposed >> > to >> > > > what it represents. >> > > > >> > > > In parquet, establishing semantic meaning is the purpose of logical >> > > types. >> > > > The logical type separates how to interpret a value from its >> physical >> > > > representation. We can represent a timestamp with INT64 or even a >> > string >> > > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and >> > > VECTOR >> > > > may have the exact same representation and performance, they have >> > > different >> > > > semantic meaning. Databases and engines across the ecosystem >> already >> > > > recognize this differentiation and have separate representations. >> The >> > > > choice to separate these isn't because the discussion is too >> tedious; >> > > > they're fundamentally different things. >> > > > >> > > > The finite requirement is what makes a vector a reflection of what >> it >> > > > represents. As we've discussed in the thread, non-finite values in >> the >> > > > representation violate the properties of a vector and the value >> ceases >> > to >> > > > be a vector. On the other hand, a 0 vector is still a vector (you >> can >> > > > argue the utility of it, but it is a valid vector). Implementations >> > can >> > > > choose what they validate or how to handle execution errors; that's >> an >> > > > implementation concern, not a requirement. Operations performed on >> a >> > > > vector depend on the context in which they are used, but the >> > > representation >> > > > doesn't need to solve scenarios where that application doesn't make >> > > sense. >> > > > >> > > > The guarantees or context needed to use a vector do not need to be >> > > inherent >> > > > in the type for it to be a useful type. We don't need the currency >> > used >> > > > for decimal or whether a floating point is a normalized value as >> part >> > of >> > > > the type to make use of those types. This conflates the use cases >> with >> > > the >> > > > type system and unnecessarily attempts to draw the context into the >> > type >> > > > definition. Similarly, you don't need to capture normalization, >> model, >> > > > etc. as part of the type for it to be useful. >> > > > >> > > > So the reason we want to add a vector type (or FINITE_VECTOR) is >> > because >> > > we >> > > > want to store vectors with the semantic meaning and requirements >> based >> > on >> > > > what they represent. >> > > > >> > > > -Dan >> > > > >> > > > >> > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <[email protected]> >> wrote: >> > > >> >> > > >> A bit late to the party, but here we go: >> > > >> >> > > >> I have the strong sentiment that the FINITE_VECTOR type as it is >> > > currently >> > > >> being discussed will not be very useful and just increase >> > implementation >> > > >> complexity. The arguments for the type are all quite weak (details >> see >> > > >> below). The FINITE part doesn't seem to yield any value and is not >> a >> > > useful >> > > >> restriction by itself. The vector (i.e. fixed length) part should >> > > rather be >> > > >> generalized instead of making a one-off thing. In general, the >> > existing >> > > >> types are already working well for uncompressed vectors. The real >> > > challenge >> > > >> that would bring real benefits is supporting quantization. This is >> > where >> > > >> you get orders of magnitude improvements to storage space and >> > processing >> > > >> speed. >> > > >> >> > > >> Let's first discuss the FINITE part. So far, the only arguments I >> have >> > > read >> > > >> for this are: >> > > >> >> > > >> 1. many other vector databases have it >> > > >> 2. vectors are the hot stuff right now, so Parquet better boast >> having >> > > >> "native" type support for vectors >> > > >> 3. by focussing on this common special case, we can make rapid >> > progress, >> > > >> while a general solution would take too long to specify >> > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in >> the >> > > >> vector >> > > >> >> > > >> the first two arguments are quite weak by themselves. Vector >> databases >> > > are >> > > >> very specialized engines. Such engines can of course be very >> > > opinionated on >> > > >> their data types. A general format like Parquet must instead weigh >> > > whether >> > > >> a special type is worth the added complexity over a more general, >> > widely >> > > >> applicable type. In this case, it is IMHO not at all. And just >> having >> > a >> > > >> type called "vector" for advertisement reasons to make Parquet >> appear >> > > more >> > > >> modern seems a non-goal in the first place. >> > > >> >> > > >> The third "time to market" argument is valid for systems that don't >> > > need a >> > > >> stable interface. It is a very dangerous attitude for formats like >> > > Parquet >> > > >> that must be stable long term and should have a consolidated, >> concise, >> > > >> non-redundant type system. For such long term storage formats, we >> need >> > > to >> > > >> take the time to build the right thing, as any shortcuts we take >> will >> > > stay >> > > >> as blemishes in the format forever. If I read the discussion >> > correctly, >> > > >> this is actually the main argument as of now: We have found that >> it is >> > > hard >> > > >> to align on how a fixed size list type should look like, so we >> want to >> > > go >> > > >> the easier route of a specialized type. This seems like a >> > > >> dangerous shortcut fueled by feature recency bias. Ask yourself: >> Would >> > > we >> > > >> want this, if Parquet already had a fixed size list type? >> > > >> >> > > >> So finally, the only technical argument is the fourth one: "It >> signals >> > > >> semantics and ensures that no bad data lands in the format". I see >> two >> > > big >> > > >> problems with this: >> > > >> >> > > >> 1. Readers *still* need to test for NaN / INF because Parquet files >> > may >> > > be >> > > >> corrupt. Nothing can stop a malicious or buggy writer from >> physically >> > > >> writing NaN or INF. *So you save no single line of code in a >> reader. >> > You >> > > >> save no branches, as you still need to handle the INF/NaN case.* So >> > this >> > > >> doesn't help with performance at all. The only difference is, that >> the >> > > >> error message changes from "found an invalid INF value in the >> data" to >> > > >> "Your parquet file seems to be corrupt". It is actually quite the >> > > opposite, >> > > >> readers and writers get more complex having to handle this new >> type. >> > > >> >> > > >> If you now argue that it's not about performance, it's about >> defined >> > > >> semantics and erroring out early in writers, then the second >> problem >> > > >> arises: >> > > >> >> > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple >> > > >> guarantees that you need for sensible vector processing. If "no >> > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that >> > > >> FINITE_VECTOR is useful, but it is by no means the only guarantee >> you >> > > need, >> > > >> instead, depending on the distance metric, you also need the >> following >> > > >> guarantees. E.g.: >> > > >> >> > > >> - No vector may be the 0 vector, otherwise, e.g., cosine >> similarity >> > is >> > > >> undefined. Cosine similarity is often used as default, so this is >> > > >> actually >> > > >> quite important in practice. >> > > >> - If we want to use a dot product, then all vectors need to >> > guarantee >> > > a >> > > >> length of 1. This would actually be quite useful for performance >> > > >> reasons. >> > > >> - The vectors must come from the same model as the vector used >> for >> > the >> > > >> search. Otherwise, the search yields a numeric result, but that >> > result >> > > >> has >> > > >> no sensible meaning. >> > > >> >> > > >> So, what use is a type that just guarantees part of the things you >> > need >> > > for >> > > >> vector search but not others? You still need additional guarantees >> in >> > > >> addition to that type. What makes INF/NaN so important in >> comparison >> > to, >> > > >> say, "no 0 vector present", that it warrants an own type, while the >> > > other >> > > >> guarantees do not? If we really want this type as a basis for >> > > cross-engine >> > > >> sharing of vectors, then it needs to give *all* guarantees. >> > > >> >> > > >> I'm not against a specialized vector type if it really carries >> *all* >> > > >> information to be useful by itself. If it carried all that >> > information, >> > > >> then a reading engine could indeed validate that only meaningful >> > vector >> > > >> searches are executed. But as long as it is just a half-baked >> thing, >> > it >> > > is >> > > >> not useful by itself and just a burden on each implementer and >> > > maintainer. >> > > >> If we need extra information anyway, why not just stick to a >> general >> > > type >> > > >> like VECTOR (without FINITE). >> > > >> >> > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) >> what >> > > >> advantage you will *really* get over just a (potentially fixed >> size) >> > > List >> > > >> of Float? As discussed, it's not performance. It's also not a >> semantic >> > > >> marker that is sufficient by itself. It's just a small hint that >> > allows >> > > >> writers to error out once they see a NaN or INF in the input, while >> > they >> > > >> will still happily write a bogus 0 vector. This seems to be of zero >> > > >> benefit. >> > > >> >> > > >> Anecdotally, we currently handle petabytes of vectors in >> production in >> > > >> Data360. We simply store them in a List of Float, with an optimized >> > > R-Level >> > > >> reader implementation that rapidly checks that all vectors are the >> > same >> > > >> length. This solution works just fine. The overhead of R-level >> > decoding >> > > is >> > > >> barely measurable. The check is super fast and R-levels are tiny in >> > > >> comparison to the payload vectors, so also the added I/O is >> > negligible. >> > > >> This is also why I'm so firm in my sentiment: We are running a >> > > production >> > > >> grade system processing trillions of vectors right now and see no >> big >> > > >> issues with the types that Parquet currently provides. >> > > >> >> > > >> As I don't just want to be a naysayer and point out problems with >> the >> > > >> current design, here are the directions that I would propose >> indeed: >> > > >> >> > > >> - a general fixed size list type. Don't do shortcuts just because >> > the >> > > >> discussion is too tedious. >> > > >> - a vector type that indeed carries enough information to be >> really >> > > >> useful for vector search by itself. (e.g., model name / version, >> > > >> optional >> > > >> guarantees for no 0 vector / normalized vector length). With >> such a >> > > >> type, >> > > >> you could really do well defined vector searches even if the >> writing >> > > >> engine >> > > >> is completely different to the reading one. >> > > >> - Standardize quantizations. This will be a real game changer for >> > > >> performance. >> > > >> >> > > >> Cheers, >> > > >> Jan >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu < >> [email protected] >> > >: >> > > >>> >> > > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST >> is a >> > > >> good >> > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more >> > > natural. >> > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first? >> > > >>> >> > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < >> > > >>> [email protected]> wrote: >> > > >>> >> > > >>>> I suggest we go with VECTOR for the finite vector to keep the >> name >> > the >> > > >>> same >> > > >>>> as other engines/type systems. If we want a non-finite one in the >> > > >> future >> > > >>> we >> > > >>>> can name that something else or we add fixed size list that will >> > serve >> > > >>> the >> > > >>>> same purpose without a special name. >> > > >>>> >> > > >>>> >> > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> >> wrote: >> > > >>>> >> > > >>>>> A logical type would: >> > > >>>>> - Describe data for users and readers to understand what the >> file >> > > >>>> contains >> > > >>>>> - Signal that repetition and definition levels will reflect a >> fixed >> > > >>>> number >> > > >>>>> of elements, which implementations can use to optimize decoding >> > > >>>>> - Guarantee that there are no null elements (because the type >> > cannot >> > > >> be >> > > >>>>> applied to a list with optional elements) >> > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity >> > > >> values >> > > >>>> so >> > > >>>>> that the vectors have a finite length and are suitable for >> distance >> > > >>>>> calculations >> > > >>>>> >> > > >>>>> The finite value restriction is a restriction on writers to >> prevent >> > > >> bad >> > > >>>>> data from being written. >> > > >>>>> >> > > >>>>> Ignoring logical types that are not understood is a rule for >> > > >> readers. A >> > > >>>>> reader can handle vectors and pass it on to another layer. For >> > > >>> instance, >> > > >>>> a >> > > >>>>> Parquet reader that doesn't support geometry passes values as >> > > >> `binary`. >> > > >>>>> That binary value can be passed to a geometry UDF, even if the >> > > >>> underlying >> > > >>>>> engine and Parquet reader doesn't have support. >> > > >>>>> >> > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass >> a >> > > >> list >> > > >>> of >> > > >>>>> floats along to a UDF. But it is still important to the user >> that >> > bad >> > > >>>>> values are not present in the data. In your example, bad values >> are >> > > >>>> vectors >> > > >>>>> that cause the centroid calculation to fail, like vectors with >> > > >> infinite >> > > >>>>> length. >> > > >>>>> >> > > >>>>> Ryan >> > > >>>>> >> > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < >> > > >>>>> [email protected]> >> > > >>>>> wrote: >> > > >>>>> >> > > >>>>>> Could we clarify the use cases for a VECTOR annotation on >> > > >>> LIST<FLOAT>? >> > > >>>>>> Specifically, what does it make possible that's impossible >> today, >> > > >> and >> > > >>>>> what >> > > >>>>>> does it make easier that's currently hard? >> > > >>>>>> >> > > >>>>>> I'm struggling to find a strong answer. I'd like every engine >> to >> > > >>> offer >> > > >>>>>> vector functions on lists of floats without needing a new >> logical >> > > >>> type. >> > > >>>>>> >> > > >>>>>> To be clear, I'm still enthusiastic about a vector type that >> > > >>> describes >> > > >>>>> how >> > > >>>>>> to unpack and scale composite representations (quantised codes >> > plus >> > > >>>>> scales, >> > > >>>>>> codebooks, etc.), in the direction ML is going. That adds >> > > >> information >> > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines >> > already >> > > >>>>>> understand the physical type, and many already offer vector >> > > >> functions >> > > >>>> on >> > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks >> are >> > > >>>> cheap >> > > >>>>>> enough that engines should do them anyway. The label also >> doesn't >> > > >> say >> > > >>>>> what >> > > >>>>>> you can do with the column. Knowing that a column holds >> embeddings >> > > >>>>> doesn't >> > > >>>>>> say how they can be used, just as zipcode and age are both >> > integers >> > > >>> but >> > > >>>>>> multiplying them is meaningless. >> > > >>>>>> >> > > >>>>>> My concern is the opposite effect. We're establishing that >> engines >> > > >>>> should >> > > >>>>>> ignore logical types they don't understand. So a tool that >> doesn't >> > > >>> know >> > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user >> compute >> > > >>>>> centroids >> > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines >> > > >>> restrict >> > > >>>>>> embedding functions to VECTOR-annotated columns, that file >> stops >> > > >>>> working >> > > >>>>>> with them, through no fault of the user. An annotation that >> > doesn't >> > > >>>>> change >> > > >>>>>> what the data is shouldn't be able to make it unusable. >> > > >>>>>> >> > > >>>>>> Am I missing a use case? >> > > >>>>>> >> > > >>>>>> Best, Will >> > > >>>>>> >> > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> >> > > >> wrote: >> > > >>>>>> >> > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to >> many >> > > >>> use >> > > >>>>>> cases. >> > > >>>>>>> It is would be great to make progress on the FINITE_VECTOR, >> and >> > > >> not >> > > >>>> to >> > > >>>>>> fit >> > > >>>>>>> every use cases into the FIXED_LENGTH_LIST. >> > > >>>>>>> >> > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> >> > > >> wrote: >> > > >>>>>>> >> > > >>>>>>>> Going through this thread again, it looks like there are a >> few >> > > >>>> things >> > > >>>>>>> that >> > > >>>>>>>> people generally agree on: >> > > >>>>>>>> >> > > >>>>>>>> - In the short term, we want to introduce a logical type or >> > > >>> types; >> > > >>>> a >> > > >>>>>>> better >> > > >>>>>>>> physical representation can be done later. We can focus on >> > > >>> logical >> > > >>>>>> types. >> > > >>>>>>>> - There is precedent in logical types for restricting values, >> > > >>> like >> > > >>>>> UTF8 >> > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also >> agree >> > > >>>> that >> > > >>>>> we >> > > >>>>>>>> could restrict vector elements to non-nested or primitive >> > > >> types. >> > > >>>>>>>> - There are many examples of databases that restrict their >> > > >> vector >> > > >>>>> types >> > > >>>>>>> to >> > > >>>>>>>> non-null, finite elements. >> > > >>>>>>>> - Some people want to have write-time guarantees that vectors >> > > >>> have >> > > >>>>> only >> > > >>>>>>>> non-null, finite elements >> > > >>>>>>>> - Some people want to extend this work to include tensors, >> > > >>> shorter >> > > >>>>>>> floating >> > > >>>>>>>> point representations, variable lengths, NaN or Infinity >> > > >>> elements, >> > > >>>>> etc. >> > > >>>>>>>> >> > > >>>>>>>> It sounds like we are nearly there, but have two halves of >> the >> > > >>>>>> community >> > > >>>>>>>> that care about different things. That just means we should >> > > >> have >> > > >>>>>> multiple >> > > >>>>>>>> types. It's entirely reasonable to want a type that allows >> NaN >> > > >>>>> values, >> > > >>>>>>> and >> > > >>>>>>>> it's also reasonable for those of us building databases, >> table >> > > >>>>>> standards, >> > > >>>>>>>> and vector indexes to care about guarantees that elements are >> > > >>>> finite. >> > > >>>>>>>> There's more than enough examples to show that this is a >> > > >>> legitimate >> > > >>>>> use >> > > >>>>>>>> case. >> > > >>>>>>>> >> > > >>>>>>>> Since I don't see the concerns of the different use cases >> > > >>> changing, >> > > >>>>> it >> > > >>>>>>>> looks like we need to find a compromise to move forward. I >> > > >>> propose >> > > >>>>> that >> > > >>>>>>> we >> > > >>>>>>>> introduce a FINITE_VECTOR type and separately a >> > > >> FIXED_LENGTH_LIST >> > > >>>> or >> > > >>>>>>>> another VECTOR type that allows infinite and NaN values. >> > > >>>>>>>> >> > > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try >> > > >> to >> > > >>>> fit >> > > >>>>>>> these >> > > >>>>>>>> use cases into a single type? What if that means not making >> > > >>>> progress? >> > > >>>>>>>> >> > > >>>>>>>> Ryan >> > > >>>>>>>> >> > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < >> > > >>>>>>>> [email protected]> wrote: >> > > >>>>>>>> >> > > >>>>>>>>> Thinking about this more, I am also fine with pulling out >> the >> > > >>>>>> “NaN/Inf” >> > > >>>>>>>>> requirement. Either into a type boolean (my preference), or >> > > >>>> putting >> > > >>>>>> no >> > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754 >> > > >> float >> > > >>>>>> numbers. >> > > >>>>>>>> If >> > > >>>>>>>>> some top-level application wants to put additional >> > > >> constraints >> > > >>> on >> > > >>>>> the >> > > >>>>>>>> data >> > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still >> do >> > > >>> so >> > > >>>>> one >> > > >>>>>>>> layer >> > > >>>>>>>>> above. It seems like this was the biggest point of >> discussion >> > > >>>>>>> remaining. >> > > >>>>>>>>> All these options work for me, and we should move forward >> > > >> with >> > > >>>> this >> > > >>>>>>>>> proposal. >> > > >>>>>>>>> >> > > >>>>>>>>> Concerning additional element types like bf16, they can be >> > > >>>>> introduced >> > > >>>>>>> as >> > > >>>>>>>>> Parquet types (at least in the current LIST-based spec >> > > >>> proposal). >> > > >>>>>>>>> >> > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a >> > > >>>>> variable >> > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I >> > > >> think >> > > >>>>> that >> > > >>>>>> a >> > > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits >> > > >> by >> > > >>>>>>>> describing >> > > >>>>>>>>> the semantics of the data, and allowing to capture future >> > > >>> changes >> > > >>>>>>> around >> > > >>>>>>>>> statistics. In that case, the length is better captured as >> an >> > > >>>> exact >> > > >>>>>>>> length >> > > >>>>>>>>> for this type specifically. I’m also not sure about other >> use >> > > >>>> cases >> > > >>>>>> for >> > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just >> > > >>> for >> > > >>>>> the >> > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough >> > > >>> value >> > > >>>> in >> > > >>>>>> my >> > > >>>>>>>>> view. >> > > >>>>>>>>> >> > > >>>>>>>>> Best, >> > > >>>>>>>>> Philipp >> > > >>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < >> > > >>>>>>>>> [email protected]> >> > > >>>>>>>>> wrote: >> > > >>>>>>>>> >> > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond >> > > >> normalized >> > > >>>>>>> embeddings >> > > >>>>>>>>> and >> > > >>>>>>>>>> engines really aren't going to be able to elide all the >> > > >>> checks >> > > >>>>>>> anyway. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :) >> > > >>>>>>>>>> >> > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't >> > > >>>> express >> > > >>>>>> this >> > > >>>>>>>> in >> > > >>>>>>>>>> Parquet. Engines can write opaque blobs they understand in >> > > >>> the >> > > >>>>>>>> metadata >> > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but >> > > >>> this >> > > >>>>>> isn't >> > > >>>>>>>>>> standardized, portable, nor interoperable. >> > > >>>>>>>>>> >> > > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift >> > > >>>> fields >> > > >>>>>> (my >> > > >>>>>>>>> pref) >> > > >>>>>>>>>> can generically support VECTOR(N) whilst also being more >> > > >>>> broadly >> > > >>>>>>> useful >> > > >>>>>>>>> for >> > > >>>>>>>>>> other use cases and data types? >> > > >>>>>>>>>> >> > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and >> > > >>>>>>> MAX_LENGTH >> > > >>>>>>>>> for >> > > >>>>>>>>>> Parquet's variable-length physcial types. These would >> > > >> count >> > > >>>>>> logical >> > > >>>>>>>>> units >> > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as >> > > >>>> LIST(N) >> > > >>>>>> with >> > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even >> > > >> different >> > > >>>>>>> engines - >> > > >>>>>>>>> can >> > > >>>>>>>>>> pick up these constraints if they support doing so. We >> > > >> don't >> > > >>>>>>> actually >> > > >>>>>>>>> need >> > > >>>>>>>>>> a logical type to express a VECTOR then. >> > > >>>>>>>>>> >> > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to >> > > >> be >> > > >>>>>>> expressed >> > > >>>>>>>> on >> > > >>>>>>>>>> UTF8 columns etc. >> > > >>>>>>>>>> >> > > >>>>>>>>>> This also seems generally useful in the future as we >> > > >> approach >> > > >>>>>>>> expressing >> > > >>>>>>>>>> other tensor data shapes and types. >> > > >>>>>>>>>> >> > > >>>>>>>>>> This won't change the mechanics: the storage type doesn't >> > > >>>> enforce >> > > >>>>>>> these >> > > >>>>>>>>>> constraints and engines must check them. It's previously >> > > >>>>>> established >> > > >>>>>>>> that >> > > >>>>>>>>>> performance is just a software problem and there is no >> > > >>>>> fundamental >> > > >>>>>>>>>> performance advantage to the fixed size types in >> > > >> Dremel-like >> > > >>>>>> formats >> > > >>>>>>>> like >> > > >>>>>>>>>> Parquet. >> > > >>>>>>>>>> >> > > >>>>>>>>>> Best, >> > > >>>>>>>>>> Will >> > > >>>>>>>>>> >> > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < >> > > >>>>>>>>> [email protected] >> > > >>>>>>>>>>> >> > > >>>>>>>>>> wrote: >> > > >>>>>>>>>> >> > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have >> > > >>> been >> > > >>>>>>>> thinking >> > > >>>>>>>>>>> about this quite a bit, and thought I would share my 2 >> > > >>> cents. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as >> > > >>>>> proposed >> > > >>>>>>> in >> > > >>>>>>>>> the >> > > >>>>>>>>>> PR >> > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> While Divjot and others note the specialization for >> > > >> UTF-8, >> > > >>>>>> Parquet >> > > >>>>>>>>>> already >> > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings >> > > >>>>>>>>>>> < >> > > >>>>>>>> >> > > >>>> >> > https://github.com/apache/parquet-format/blob/master/README.md#types >> > > >>>>>>>>>> . >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new >> > > >>>>>>> restriction >> > > >>>>>>>>> on >> > > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE >> > > >>>>> columns >> > > >>>>>>>> have >> > > >>>>>>>>>>> never had such a rule. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> It would also not match how the systems that actually >> > > >>> produce >> > > >>>>>> these >> > > >>>>>>>>>> vectors >> > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark, >> > > >>>> DuckDB, >> > > >>>>>>>> NumPy, >> > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and >> > > >>>> OpenBLAS. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an >> > > >> insert >> > > >>>>>> policy. >> > > >>>>>>>>> While >> > > >>>>>>>>>> I >> > > >>>>>>>>>>> think Parquet as a general-purpose format should allow >> > > >> them >> > > >>>> out >> > > >>>>>> of >> > > >>>>>>>> the >> > > >>>>>>>>>> box. >> > > >>>>>>>>>>> It is a user system decision to disallow writing these >> > > >>> values >> > > >>>>> or >> > > >>>>>> to >> > > >>>>>>>>> throw >> > > >>>>>>>>>>> errors when encountering them. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector >> > > >>>> math. >> > > >>>>>>>> Finite >> > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude = >> > > >>>>>> ||[FLOAT_MAX, >> > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows >> > > >> to >> > > >>>>> +Inf. >> > > >>>>>>>>> PyTorch >> > > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm() >> > > >> is >> > > >>>> inf >> > > >>>>>> in >> > > >>>>>>>>> fp32. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while >> > > >>> the >> > > >>>>>>> VECTOR >> > > >>>>>>>>>> itself >> > > >>>>>>>>>>> was required to be finite. The ban only forbids storing >> > > >>>>> specials >> > > >>>>>> in >> > > >>>>>>>> the >> > > >>>>>>>>>>> coordinates, and does not prevent even simple downstream >> > > >>>>>>> computations >> > > >>>>>>>>>> from >> > > >>>>>>>>>>> producing them. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional >> > > >>> bool >> > > >>>>>>>>>>> VectorType.finite (default false). Engines that want >> > > >>>>>> pgvector-like >> > > >>>>>>>>>>> rejection can require the flag = true. Engines that use >> > > >>>> NumPy / >> > > >>>>>>>>> PyTorch / >> > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the >> > > >>> same >> > > >>>>>>>> reason, I >> > > >>>>>>>>>> do >> > > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Best, >> > > >>>>>>>>>>> Sebastian >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < >> > > >>>>>>>>>>> [email protected]> >> > > >>>>>>>>>>> wrote: >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> I agree with moving forward with the proposal and >> > > >>> renaming >> > > >>>>>> VECTOR >> > > >>>>>>>> to >> > > >>>>>>>>>>>> FINITE_VECTOR or something >> > > >>>>>>>>>>>> similar if others feel that better communicates its >> > > >>>> intent. I >> > > >>>>>> see >> > > >>>>>>>>> value >> > > >>>>>>>>>>> in >> > > >>>>>>>>>>>> having a vector type that guarantees >> > > >>>>>>>>>>>> its values are mathematically-correct vectors for a few >> > > >>>>>> reasons: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the >> > > >>> STRING >> > > >>>>>>> logical >> > > >>>>>>>>> type >> > > >>>>>>>>>>>> mandating UTF-8. >> > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for >> > > >> detecting >> > > >>>> NaN >> > > >>>>>> or >> > > >>>>>>>>> +-inf >> > > >>>>>>>>>>>> values as all statistics are optional. >> > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring >> > > >> that >> > > >>>>>> writers >> > > >>>>>>>>> emit >> > > >>>>>>>>>>>> stats for specific logical types. >> > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow >> > > >> through >> > > >>>> the >> > > >>>>>>> engine >> > > >>>>>>>>> and >> > > >>>>>>>>>>>> block them for specific functionality. >> > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't >> > > >> think >> > > >>>>> this >> > > >>>>>> is >> > > >>>>>>>> as >> > > >>>>>>>>>> easy >> > > >>>>>>>>>>>> as it sounds and Lance has run >> > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples). >> > > >>> Their >> > > >>>>>>> Python >> > > >>>>>>>>> [3] >> > > >>>>>>>>>>> and >> > > >>>>>>>>>>>> Rust [4] APIs default to erroring on >> > > >>>>>>>>>>>> vectors that contain NaN values. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I >> > > >>>> don't >> > > >>>>>>> think >> > > >>>>>>>>>> this >> > > >>>>>>>>>>>> implies the lack of domain-specific types. >> > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet >> > > >> would >> > > >>> be >> > > >>>>>>>>> considered >> > > >>>>>>>>>> a >> > > >>>>>>>>>>>> good fit. I also fully agree with >> > > >>>>>>>>>>>> continuing work on a general fixed-size list type if we >> > > >>>>> believe >> > > >>>>>>>> there >> > > >>>>>>>>>> are >> > > >>>>>>>>>>>> non-vector use cases that would benefit >> > > >>>>>>>>>>>> from such guarantees at the storage layer. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648 >> > > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607 >> > > >>> (see >> > > >>>>>>> point 2 >> > > >>>>>>>>> in >> > > >>>>>>>>>>>> "root >> > > >>>>>>>>>>>> cause") >> > > >>>>>>>>>>>> [3] >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> > >> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 >> > > >>>>>>>>>>>> [4] >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> > >> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via >> > > >>> dev < >> > > >>>>>>>>>>>> [email protected]> wrote: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>>> I also favor moving forward with the proposal as is. >> > > >> I >> > > >>>> see >> > > >>>>>>> strong >> > > >>>>>>>>>> value >> > > >>>>>>>>>>>> and >> > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree >> > > >> with >> > > >>>> the >> > > >>>>>>>>>>> restrictions >> > > >>>>>>>>>>>>> around NaN/Inf values and element types in general. >> > > >>>>> Precedent >> > > >>>>>>>>> exists >> > > >>>>>>>>>> in >> > > >>>>>>>>>>>>> other types that restrict which physical type values >> > > >>> are >> > > >>>>>>>> logically >> > > >>>>>>>>>>> valid >> > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be >> > > >> WKB-encoded). >> > > >>>> A >> > > >>>>>>>> general >> > > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes >> > > >>>> sense >> > > >>>>> to >> > > >>>>>>>>>> separate >> > > >>>>>>>>>>>> the >> > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see >> > > >>> how >> > > >>>>> the >> > > >>>>>>> two >> > > >>>>>>>>>> types >> > > >>>>>>>>>>>>> might diverge further in the future regarding >> > > >>> statistics >> > > >>>> or >> > > >>>>>>>>>> encodings. >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it >> > > >>>> seems >> > > >>>>>>> like >> > > >>>>>>>>> the >> > > >>>>>>>>>>>> VECTOR >> > > >>>>>>>>>>>>> name is already used across other engines mentioned >> > > >> in >> > > >>>> this >> > > >>>>>>>> thread. >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> Best, >> > > >>>>>>>>>>>>> Philipp >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < >> > > >>>> [email protected]> >> > > >>>>>>>> wrote: >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should >> > > >>> treat >> > > >>>>> the >> > > >>>>>>>>> general >> > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout, >> > > >> and >> > > >>>>>> separate >> > > >>>>>>>> it >> > > >>>>>>>>>> from >> > > >>>>>>>>>>>> the >> > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML >> > > >>>> tensor) >> > > >>>>>>>> logical >> > > >>>>>>>>>>> type. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> Currently, the immediate need is to define a >> > > >>>> specialized >> > > >>>>>>> VECTOR >> > > >>>>>>>>>> type >> > > >>>>>>>>>>> so >> > > >>>>>>>>>>>>>> that different engines and formats have a common >> > > >>>> semantic >> > > >>>>>>>>> contract >> > > >>>>>>>>>>> for >> > > >>>>>>>>>>>>>> interoperability. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> If the main point of contention is just the >> > > >>>> NaN/Inf/null >> > > >>>>>>>>>> limitations, >> > > >>>>>>>>>>>>> maybe >> > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to >> > > >> the >> > > >>>> name >> > > >>>>>>> (like >> > > >>>>>>>>>>> Andrew >> > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) >> > > >>> to >> > > >>>>>> avoid >> > > >>>>>>>>>>> confusion >> > > >>>>>>>>>>>>> with >> > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could >> > > >>>> introduce a >> > > >>>>>>>> boolean >> > > >>>>>>>>>>> flag >> > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed >> > > >>>>>> earlier. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < >> > > >>>>>>>>>> [email protected]> >> > > >>>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> I am a little confused about the points of >> > > >>>> disagreement >> > > >>>>>>> here. >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for >> > > >> the >> > > >>>>>>> specific >> > > >>>>>>>>> use >> > > >>>>>>>>>>> case >> > > >>>>>>>>>>>>> of >> > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used >> > > >> for >> > > >>>>>> vector >> > > >>>>>>>>>>>> embeddings? >> > > >>>>>>>>>>>>>> If >> > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the >> > > >> name >> > > >>>>>>>>>> (FINITE_VECTOR? >> > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> It seems like there is precedent for defining >> > > >>> logical >> > > >>>>>> types >> > > >>>>>>>>> that >> > > >>>>>>>>>>>>> specify >> > > >>>>>>>>>>>>>>> certain physical types and restrict their >> > > >> contents >> > > >>> to >> > > >>>>>> help >> > > >>>>>>>>>> readers >> > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON; >> > > >>>>>> Defining >> > > >>>>>>>> one >> > > >>>>>>>>>> for >> > > >>>>>>>>>>>>>> vector >> > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> Andrew >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> [1]: >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> > >> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < >> > > >>>>>>>>> [email protected]> >> > > >>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format. >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits >> > > >>>> their >> > > >>>>>>>> usage, >> > > >>>>>>>>>> but >> > > >>>>>>>>>>>>> we're >> > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format >> > > >>> that >> > > >>>>> aims >> > > >>>>>>> to >> > > >>>>>>>> be >> > > >>>>>>>>>>>> broadly >> > > >>>>>>>>>>>>>>>>> applicable. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> I agree that the format is intended to be >> > > >> general >> > > >>>>>>> purpose, >> > > >>>>>>>>> but >> > > >>>>>>>>>>>> types >> > > >>>>>>>>>>>>>>> within >> > > >>>>>>>>>>>>>>>> the format define specific representation and >> > > >>>>> semantics >> > > >>>>>>> to >> > > >>>>>>>>>>> achieve >> > > >>>>>>>>>>>>> that >> > > >>>>>>>>>>>>>>>> goal. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and >> > > >>>> it's >> > > >>>>>>> quite >> > > >>>>>>>>>>> trivial >> > > >>>>>>>>>>>> to >> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using >> > > >>> e.g. >> > > >>>>>> NumPy >> > > >>>>>>>> or >> > > >>>>>>>>>>> Pandas >> > > >>>>>>>>>>>>> (or >> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific >> > > >>>>> computing >> > > >>>>>>>>>> software). >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> The justification for restricting non-finite >> > > >>>> values >> > > >>>>> is >> > > >>>>>>>>>> directly >> > > >>>>>>>>>>>>>> related >> > > >>>>>>>>>>>>>>> to >> > > >>>>>>>>>>>>>>>> the semantic meaning of a vector. Vectors have >> > > >>>>>> specific >> > > >>>>>>>>>>>>>> characteristics >> > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). >> > > >>> Non-finite >> > > >>>>>>> values >> > > >>>>>>>>> are >> > > >>>>>>>>>>>>> invalid >> > > >>>>>>>>>>>>>>> and >> > > >>>>>>>>>>>>>>>> break those characteristics. I'd liken it to >> > > >>>> storing >> > > >>>>>>>>> non-utf8 >> > > >>>>>>>>>>>> values >> > > >>>>>>>>>>>>>> in >> > > >>>>>>>>>>>>>>> a >> > > >>>>>>>>>>>>>>>> string and still calling it a string. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed >> > > >>>> length >> > > >>>>>>> list, >> > > >>>>>>>>>> which >> > > >>>>>>>>>>>> is >> > > >>>>>>>>>>>>>> more >> > > >>>>>>>>>>>>>>>> general and does not carry the same semantic >> > > >>>> meaning. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all >> > > >>> similarly >> > > >>>>>> shaped >> > > >>>>>>>> use >> > > >>>>>>>>>>> cases >> > > >>>>>>>>>>>>>> into >> > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list). >> > > >> Vector >> > > >>>> is >> > > >>>>> a >> > > >>>>>>> well >> > > >>>>>>>>>>>>> established >> > > >>>>>>>>>>>>>>>> data type at this point and aligns with the >> > > >>> general >> > > >>>>>>> purpose >> > > >>>>>>>>>>> nature >> > > >>>>>>>>>>>> of >> > > >>>>>>>>>>>>>> the >> > > >>>>>>>>>>>>>>>> format (many other formats and databases >> > > >> natively >> > > >>>>>> support >> > > >>>>>>>>>> this). >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> -Dan >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine >> > > >> Pitrou < >> > > >>>>>>>>>>>> [email protected]> >> > > >>>>>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a >> > > >> écrit >> > > >>> : >> > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the >> > > >>>> clear >> > > >>>>>>>>> use-case >> > > >>>>>>>>>>> for >> > > >>>>>>>>>>>>>>>> non-finite >> > > >>>>>>>>>>>>>>>>>> values within a vector type. There has >> > > >> been a >> > > >>>> lot >> > > >>>>>> of >> > > >>>>>>>>>>>> theoretical >> > > >>>>>>>>>>>>>>>>> discussion >> > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings >> > > >>>> currently >> > > >>>>>>>> produce >> > > >>>>>>>>>>> these >> > > >>>>>>>>>>>>>>> vectors >> > > >>>>>>>>>>>>>>>>> and >> > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines? >> > > >>> I'd >> > > >>>>>> like >> > > >>>>>>> to >> > > >>>>>>>>>> limit >> > > >>>>>>>>>>>>>>>> introducing >> > > >>>>>>>>>>>>>>>>>> types that have a wide range of values >> > > >> which >> > > >>>> are >> > > >>>>>>> simply >> > > >>>>>>>>> are >> > > >>>>>>>>>>>>> failure >> > > >>>>>>>>>>>>>>>> modes >> > > >>>>>>>>>>>>>>>>>> of different flavors >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and >> > > >>>> it's >> > > >>>>>>> quite >> > > >>>>>>>>>>> trivial >> > > >>>>>>>>>>>> to >> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using >> > > >>> e.g. >> > > >>>>>> NumPy >> > > >>>>>>>> or >> > > >>>>>>>>>>> Pandas >> > > >>>>>>>>>>>>> (or >> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific >> > > >>>>> computing >> > > >>>>>>>>>> software). >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector >> > > >>> is >> > > >>>> a >> > > >>>>>> IETF >> > > >>>>>>>>>>> standard >> > > >>>>>>>>>>>> or >> > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8 >> > > >> (where >> > > >>>> the >> > > >>>>>>>> values >> > > >>>>>>>>>> are >> > > >>>>>>>>>>>>>> expected >> > > >>>>>>>>>>>>>>>>> to be valid according to the respective >> > > >> specs) >> > > >>> do >> > > >>>>> not >> > > >>>>>>>>> apply. >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better >> > > >> chance >> > > >>>> of >> > > >>>>>>>>> converging >> > > >>>>>>>>>>> if >> > > >>>>>>>>>>>> we >> > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of: >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data >> > > >>> in >> > > >>>>>>> Parquet >> > > >>>>>>>>>> (with >> > > >>>>>>>>>>> a >> > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types) >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> 2. representing the information that a given >> > > >>>> column >> > > >>>>>> (or >> > > >>>>>>>>>> column >> > > >>>>>>>>>>>>> chunk, >> > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own >> > > >>> right, >> > > >>>>> but >> > > >>>>>> do >> > > >>>>>>>> not >> > > >>>>>>>>>>> have >> > > >>>>>>>>>>>> to >> > > >>>>>>>>>>>>>> be >> > > >>>>>>>>>>>>>>>>> conflated and bundled together. >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Regards >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Antoine. >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> > > >> > >> >
