Jan, I agree that we _could_ model vectors as an orthogonal finiteness property for elements in a fixed-length list. I don't know how useful that would be. I've not had anyone ask for that, and I'm not aware of databases with finite floating point types. I think it makes more sense to move forward with the existing FINITE_VECTOR that is proposed.
For the logical type argument, it sounds like the issue is that you want to fall back to readers treating vectors as LIST. I don't see a good way to make that happen, but it also isn't needed by the table format use case because we are introducing this as a new type. Do you have a use case where you are going to mix FINITE_VECTOR with LIST files and this will cause a problem? I don't think that we've had an expectation for logical types in the past that they produce a particular in-memory representation. Maybe we could allow setting both `LIST` and `FINITE_VECTOR` logical types at the same time? This kind of compatibility hasn't been a priority as far as I'm aware, but if it is important to you we can consider it. Ryan On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <[email protected]> wrote: > assuming what will be proposed is a new logical type FINITE_VECTOR that > annotates the same column that would have been annotated as LIST before: > > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness is part > of the vector type. Isn't the finiteness rather a property of the contained > floats? Such a modeling would be more composable, as it would for example > also allow to have normal (non-vector) float columns that are restricted to > be finite. So, if we do think that we want finiteness in the type system, I > would rather cut it into two parts: A fixed size vector concept, and a new > way of labeling float columns finite. The vector type would then just boil > down to be a fixed size list / we could just give the list type an optional > fixed size field. Then, the column contained within it would be a float > tagged as finite. > > It was mentioned that a logical type annotating a list of float would be > forward compatible, but - depending on the modeling - that is not > necessarily true: as logical types in Parquet do not support subtyping, a > column cannot be a list of float and a (finite) vector at the same time, > but only one of the two. It depends on how the type will be modeled in > thrift in the end, but if it is a new union member of `LogicalType`, then > legacy engines will not understand that such a vector can also just be > interpreted as a list of floats. They will just see a nested column type of > a logical type they don't understand, so they will likely reject reading > this column (at least that's what the engine I'm maintaining would be > doing). > > In contrast, if we just give the list type an optional fixed size and float > an optional finiteness, then the resulting thrift is actually fully forward > compatible: Legacy engines will just ignore the fixed length and treat the > list as variable length and they will ignore the finiteness and treat the > element type as float. > > (A side note, which most people here may already be aware of, but it's good > to repeat it: The not-NULL property doesn't need new type support at all. > The normal type system of list already allows setting the contained element > type to be optional or required. If it is set to be required, then NULLs > are not allowed.) > > Cheers, > Jan > > > > > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer < > [email protected]>: > > > I support proceeding with the two types. I honestly think the cost of > > adding new logical types is overestimated. If we have as many interested > > parties as have appeared in this thread we already know there is a large > > group of engineers ready to do the work and engines ready to consume (and > > produce) the type. The fact that implementations currently use > heuristics, > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC > > Glasgow) or custom annotations to achieve the same performance benefits > > makes me think we should just create the type and move on. > > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <[email protected]> wrote: > > > > > An annotation on list<float> is useful because: > > > - We want to correctly annotate what is being stored > > > - We want to unlock optimizations when each non-null list has the same > > > number of elements > > > - We want to have writer guarantees > > > > > > I don't think that logical type compatibility is a good argument. We > want > > > to avoid a case where we have to treat all binary like a string. And > > > because we are introducing this in table formats after standardizing it > > > here, we can do that reliably. > > > > > > As I said in my last email, I think there's a difference between > > finiteness > > > and general "invariants" you might want. Vectors aren't vectors if they > > > don't have a length and a direction. Other invariants don't work the > same > > > way. A zero-length vector is still a valid vector and can be used, just > > not > > > for certain distance metrics. > > > > > > I also want to clarify that when we looked into this more closely and > > > recognized the mismatch between the two separate use cases, I offered > to > > > split out the finite vector work so that we didn't cause any headaches > > for > > > people who are interested in fixed-length list. It sounds like not > > > splitting out the use cases is causing a lot of headache. Vectors for > our > > > use case need a finiteness guarantee at write time, and fixed-length > > lists > > > need to store NaN and other values. > > > > > > It's okay to have two types and we should for two different use cases. > > > Otherwise, we are forcing one to make an unnecessary compromise. I > don't > > > want to force everyone to exclude NaN and infinite values from > > fixed-length > > > lists. Seems reasonable that we should also be able to have vectors > that > > > exclude those values, since that is how they are used and we have to > also > > > guarantee it for exchanging data with other existing databases. > > > > > > Ryan > > > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev < > > > [email protected]> > > > wrote: > > > > > > > It reads well. Three small fixes: > > > > > > > > 1. **"back in May or so"** is ambiguous. The > > fixed-size-list-for-tensors > > > > framing is from Rok's opening post in May 2024. The May 2026 design > doc > > > was > > > > already mostly about performance and layout. "Originally" avoids > anyone > > > > correcting the date. > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite > > > > right. The NaN-for-missing behaviour comes from `VectorAssembler` and > > > > `Imputer` and applies to Spark ML vectors generally, dense or sparse. > > > > "Spark's feature vectors, dense and sparse," fixes it. > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and > > > > DuckDB and pretty much…" has a doubled "and". > > > > > > > > The last sentence of the invariants paragraph could be clearer. > > "Although > > > > that doesn't mean readers shouldn't check on read" has a double > > negative > > > > and is easy to misread. "Though careful readers will still check on > > read" > > > > is simpler. > > > > > > > > With those applied: > > > > > > > > --- > > > > > > > > Hi all, > > > > > > > > The scope of this discussion keeps moving :). IIRC it originally was > > > about > > > > fixed-size lists for ML tensors, became a performance question, and > is > > > now > > > > about whether vectors are mathematically valid. Each step made sense, > > > but I > > > > don't think we have consensus on where we've ended up? > > > > > > > > I think there is much useful work in what we left behind. Many ML > > vectors > > > > aren't embeddings for search. E.g. Spark's feature vectors, dense and > > > > sparse, routinely use NaN to mark missing values, and quantised > formats > > > > pack codes with scales in ways Parquet can't describe today. > > ML-oriented > > > > formats like Lance are moving towards tensors, quantisation and > > indexes. > > > A > > > > type that told engines how to interpret that kind of data would > enable > > > new > > > > things. An annotation on LIST<FLOAT> doesn't. > > > > > > > > My big concern with the current focus on embeddings needing a logical > > > type > > > > is compatibility. Today LIST<FLOAT> is the common format for > > embeddings, > > > > and Spark, BigQuery, DuckDB and pretty much every other library I can > > > think > > > > of offer distance functions on plain lists, so any engine can use any > > > file. > > > > If engines start requiring a VECTOR annotation, existing files and > > > anything > > > > rewritten by a tool that doesn't know the type stop working. If they > > > don't > > > > require it, the annotation adds nothing. We've seen this before with > > > > strings written without the UTF8 annotation, which engines still have > > to > > > > cope with years later. > > > > > > > > On invariants, I agree some users may want the writer to check what > > they > > > > write as they write it, but that can be a writer option rather than a > > > type. > > > > Readers that want evidence could get it from extra statistics or > > > metadata, > > > > though careful readers will still check on read. > > > > > > > > If a finite fixed-length vector contract is useful to table formats, > > > > perhaps Iceberg is the place for it, with Parquet storing plain > lists. > > > I'd > > > > raise the same worry there, though: if tables start rejecting > untagged > > > > columns, we've just moved the split up a layer. > > > > > > > > Best, > > > > Will > > > > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <[email protected]> wrote: > > > > > > > > > I agree with much of what Dan said, but I'll add some to this > thread > > as > > > > > well. > > > > > > > > > > First, I want to address the arguments that were pointed out: > > > > > 1. Other vector databases have finiteness requirements > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the > > bandwagon > > > > > 3. A general solution will take too long, finiteness will make > faster > > > > > progress > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will be in > the > > > > > vector > > > > > > > > > > I don't think these represent our actual motivation or position, > > which > > > > > makes them function as a straw-man. This is probably not what was > > > > > intended, but I want to make sure that the misunderstanding > > > > > doesn't continue. For one thing, we aren't saying that we need > finite > > > > > vectors because they are "hot stuff". > > > > > > > > > > We want to build tools that use them on top of Parquet. I think it > > > should > > > > > also be clear that we aren't advocating for finite elements blindly > > > just > > > > > because other engines do (cargo cult design!). There are real > > > > > interoperability issues if we choose to go a different direction so > > > this > > > > is > > > > > a "Chesterton's Fence" situation. It's important to understand why > > > > > those databases > > > > > made that choice. I've also pointed this out as confirmation that > > we're > > > > > going down a reasonable and established path. > > > > > > > > > > The third argument sounds like a simple misunderstanding. We're > > > pursuing > > > > a > > > > > logical type right now because that's forward compatible and we > want > > to > > > > > label the data properly and set expectations for the type. Physical > > > type > > > > > changes break forward compatibility and need to be released in a > new > > > > format > > > > > version . I think this is the source of the confusion since it has > > been > > > > in > > > > > the same discussion, but the choice about finiteness is separate. > > > > > > > > > > That brings us to the last argument, whether there is value in > > writers > > > > > participating in guarantees. I think there's value in adding > > > requirements > > > > > for writers. It's up to readers how they want to use those > > > requirements, > > > > > but a writer requirement allows a policy of "garbage in garbage > out" > > > > rather > > > > > than being forced to handle bad data. Why would a reader need to > run > > > > > validation on every element just in case a writer had a bug? And > why > > > > should > > > > > the potential for writer bugs mean that it has to be the reader's > > > > problem? > > > > > If a writer produces bad data, we can absolutely reject it and tell > > the > > > > > writer to fix the bug. > > > > > > > > > > I'm also puzzled by the assertion that writer guarantees have > value, > > > but > > > > > only if there are other, more specific guarantees in addition. This > > > seems > > > > > to cede the point that a contract between readers and writers makes > > > > sense, > > > > > but only if it is a _specific_ contract. But the proposed additions > > > don't > > > > > make sense in the same way. > > > > > > > > > > Take, for example, the suggestion to also disallow zero-length > > vectors. > > > > At > > > > > a glance this seems similar, but I think it assumes a specific > > context > > > > and > > > > > is better suited to be a constraint for some use cases rather than > > part > > > > of > > > > > what is considered valid for the type. That's because zero-length > > > vectors > > > > > are valid vectors with a length. You can calculate Euclidean or > > > Manhattan > > > > > distance and use them in many contexts. They don't work if you're > > > > > normalizing to a length of 1, but it's a choice to normalize and > use > > a > > > > dot > > > > > product to calculate cosine distance. And at this point, we're > > talking > > > > > about the usefulness of restrictions for the type, not a Platonic > > > ideal. > > > > > > > > > > I think the restriction that makes the most sense is that vectors > > have > > > a > > > > > length and can be compared in distance calculations. 0-length > vectors > > > are > > > > > valid because they point somewhere and have a length. We shouldn't > > > > restrict > > > > > uses to only cosine distance over normalized vectors. Those are use > > > case > > > > > dependent choices and there are other valid ones. I also don't > think > > > that > > > > > the model used to produce a vector is part of the vector type, just > > > like > > > > > the method used to produce a metric doesn't define its type: cosine > > > > > similarity produces values between -1 and 1, but we store it in a > > > > `float`, > > > > > not a `float(cosine)`. > > > > > > > > > > Why does it make sense to have a finiteness restriction as part of > > the > > > > > type? Because that's what a (mathematical) vector is. A simple > search > > > > gives > > > > > a pretty common definition: "A vector is an object that has both > > > > magnitude > > > > > and a direction". If a dimension/element is NaN, null, or infinite, > > the > > > > > vector doesn't point anywhere and it doesn't have a magnitude. Not > > only > > > > can > > > > > it not be used, it doesn't meet the most basic definition of the > > type. > > > > > > > > > > Ryan > > > > > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <[email protected]> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Daniel, > > > > > > > > > > > > alright, so no physical optimizations intended. We only want > this > > > for > > > > > > semantic reasons, not for any significant physical optimization. > If > > > > we're > > > > > > all on the same page here, that is good 👍. > > > > > > > > > > > > To phrase my remaining problem with this more concretely: > > > > > > *Let's say we add this type. What cool new stuff can we now do > that > > > we > > > > > > couldn't do before?* > > > > > > That is the litmus test that a type has to pass to be useful > IMHO. > > > And > > > > > here > > > > > > I can't come up with a tangible use case that wasn't possible > > before. > > > > > > Especially if we one day have fixed size lists in Parquet, the > > > > remaining > > > > > > niche for this type becomes so small, that the added complexity > > isn't > > > > > worth > > > > > > the benefit IMHO. > > > > > > > > > > > > As I guess most people on here are active maintainers of a > Parquet > > > > > > implementation: Concretely, what would you be using this type for > > if > > > we > > > > > had > > > > > > it? Would you allow math only on fields tagged with this type? > What > > > if > > > > > that > > > > > > math yields INF or NaN? Not allowed? > > > > > > > > > > > > Especially if we also discuss having both VECTOR and > FINITE_VECTOR. > > > How > > > > > > will you guys wire this up with the type systems of your engines? > > Do > > > > your > > > > > > engines also have two types here? Do you type-decay a > FINITE_VECTOR > > > to > > > > a > > > > > > VECTOR once someone does a multiplication that could yield INF? > By > > > > > itself, > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a type > > > system > > > > > > perspective, it comes with a lot of complexity. Do the benefits > > > really > > > > > > outweigh the complexity here? That's why I would rather have the > > bare > > > > > > minimum: A fixed size array (list) of floats or at least just a > > > vector > > > > > > without finite. Then all these detailed questions go away. > > > > > > > > > > > > I'll take this one step further... Why is there any debate about > > this > > > > at > > > > > > > all? Why can't a type simply be added to the *specification* as > > an > > > > > > > administrative detail and be done? > > > > > > > > > > > > > > > > > > Andrew, > > > > > > > > > > > > Parquet is supposed to be a lingua franca of data. The > > specification > > > is > > > > > the > > > > > > authoritative truth for this language and all implementations are > > > > > expected > > > > > > to implement as much as possible from this spec, so anything in > the > > > > spec > > > > > > does causes implementation effort. So just adding anything to > this > > > spec > > > > > > without debate is not a good idea. It is even the other way > around: > > > We > > > > > need > > > > > > to have way higher rigor with the spec than we need to have with > > any > > > > > > implementations. Implementations can be changed, the spec needs > to > > be > > > > > > stable. > > > > > > > > > > > > A tag-only type that doesn't come with any restrictions can be > > added > > > > with > > > > > > little effort. But here we want to tie restrictions to it (no > > > NaN/Inf), > > > > > so > > > > > > we would expect implementations to enforce these. So there is > some > > > > > > implementation effort expected here. > > > > > > > > > > > > Also, even if we only wanted a tag type, I would still argue that > > you > > > > > > shouldn't add those without a debate. What if A wants XML, B > wants > > > > YAML, > > > > > C > > > > > > wants quaternions, D wants address, E wants phone number. If we > > just > > > > > added > > > > > > any types that are "just tags", Parquet logical types would just > > > > become a > > > > > > trash dump of any type that anyone ever thought to be useful. So > > > > rigorous > > > > > > assessment of each new type candidate is necessary. > > > > > > > > > > > > Finally, many engines couple parquet with another type system > > (e.g., > > > > > Arrow > > > > > > or SQL). So even with a tag only type, you at least still have > the > > > > > > implementation burden of wiring this up with your engine's type > > > system. > > > > > > > > > > > > So, just because logical types are "just" logical, that doesn't > > mean > > > we > > > > > can > > > > > > add them without any rigor. > > > > > > > > > > > > > > > > > > A few naive questions: > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow > efficient > > > > > > > encodings like ALP? > > > > > > > 2. If the physical representation allows values that make > > encodings > > > > > like > > > > > > > ALP impossible, can a tighter Logical type allow different > > > encodings? > > > > > > > 3. Can we RLE or binpack a vector? > > > > > > > > > > > > > > > > > > The simple general answer here is that all these things do not > > > require > > > > a > > > > > > logical type. Writers can already do any such optimization if the > > > > actual > > > > > > data contains no NaNs & Infs. All these physical optimizations > just > > > > > require > > > > > > no actual NaNs/Infs to be present in a page, they don't need a > > > > guarantee > > > > > > that the whole file won't contain any. > > > > > > > > > > > > Cheers, > > > > > > Jan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb < > > [email protected] > > > >: > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > Glad to see a Vector type being added. I have a few use cases I > > > want > > > > to > > > > > > > pursue once we have an agreed upon definition. > > > > > > > > > > > > > > In regards to that definition discussion. I feel comfortable > with > > > the > > > > > > idea > > > > > > > of the semantic meaning Dan raises though I can’t shake a > slight > > > > > > discomfort > > > > > > > (maybe FOMO) with limiting values in the physical definition. > > > > > > > > > > > > > > A few naive questions: > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow > efficient > > > > > > > encodings like ALP? > > > > > > > 2. If the physical representation allows values that make > > encodings > > > > > like > > > > > > > ALP impossible, can a tighter Logical type allow different > > > encodings? > > > > > > > 3. Can we RLE or binpack a vector? > > > > > > > > > > > > > > Best Regards, > > > > > > > Kurtis C. Wright > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > Hey Jan, > > > > > > > > > > > > > > > > There's a lot that I disagree with respect to the framing of > > > these > > > > > > > > arguments and how we should be thinking about introducing > > vector > > > > as a > > > > > > > > logical type. There's a repeated tendency to let the > physical > > > > > > > > representation or performance drive discussion around types > as > > > > > opposed > > > > > > to > > > > > > > > what it represents. > > > > > > > > > > > > > > > > In parquet, establishing semantic meaning is the purpose of > > > logical > > > > > > > types. > > > > > > > > The logical type separates how to interpret a value from its > > > > physical > > > > > > > > representation. We can represent a timestamp with INT64 or > > even > > > a > > > > > > string > > > > > > > > value, so why have a type for it? Even though a > > FIXED_SIZED_LIST > > > > and > > > > > > > VECTOR > > > > > > > > may have the exact same representation and performance, they > > have > > > > > > > different > > > > > > > > semantic meaning. Databases and engines across the ecosystem > > > > already > > > > > > > > recognize this differentiation and have separate > > representations. > > > > > The > > > > > > > > choice to separate these isn't because the discussion is too > > > > tedious; > > > > > > > > they're fundamentally different things. > > > > > > > > > > > > > > > > The finite requirement is what makes a vector a reflection of > > > what > > > > it > > > > > > > > represents. As we've discussed in the thread, non-finite > > values > > > in > > > > > the > > > > > > > > representation violate the properties of a vector and the > value > > > > > ceases > > > > > > to > > > > > > > > be a vector. On the other hand, a 0 vector is still a vector > > > (you > > > > > can > > > > > > > > argue the utility of it, but it is a valid vector). > > > > Implementations > > > > > > can > > > > > > > > choose what they validate or how to handle execution errors; > > > that's > > > > > an > > > > > > > > implementation concern, not a requirement. Operations > > performed > > > > on a > > > > > > > > vector depend on the context in which they are used, but the > > > > > > > representation > > > > > > > > doesn't need to solve scenarios where that application > doesn't > > > make > > > > > > > sense. > > > > > > > > > > > > > > > > The guarantees or context needed to use a vector do not need > to > > > be > > > > > > > inherent > > > > > > > > in the type for it to be a useful type. We don't need the > > > currency > > > > > > used > > > > > > > > for decimal or whether a floating point is a normalized value > > as > > > > part > > > > > > of > > > > > > > > the type to make use of those types. This conflates the use > > > cases > > > > > with > > > > > > > the > > > > > > > > type system and unnecessarily attempts to draw the context > into > > > the > > > > > > type > > > > > > > > definition. Similarly, you don't need to capture > > normalization, > > > > > model, > > > > > > > > etc. as part of the type for it to be useful. > > > > > > > > > > > > > > > > So the reason we want to add a vector type (or FINITE_VECTOR) > > is > > > > > > because > > > > > > > we > > > > > > > > want to store vectors with the semantic meaning and > > requirements > > > > > based > > > > > > on > > > > > > > > what they represent. > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis < > [email protected]> > > > > > wrote: > > > > > > > >> > > > > > > > >> A bit late to the party, but here we go: > > > > > > > >> > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR type as > it > > is > > > > > > > currently > > > > > > > >> being discussed will not be very useful and just increase > > > > > > implementation > > > > > > > >> complexity. The arguments for the type are all quite weak > > > (details > > > > > see > > > > > > > >> below). The FINITE part doesn't seem to yield any value and > is > > > > not a > > > > > > > useful > > > > > > > >> restriction by itself. The vector (i.e. fixed length) part > > > should > > > > > > > rather be > > > > > > > >> generalized instead of making a one-off thing. In general, > the > > > > > > existing > > > > > > > >> types are already working well for uncompressed vectors. The > > > real > > > > > > > challenge > > > > > > > >> that would bring real benefits is supporting quantization. > > This > > > is > > > > > > where > > > > > > > >> you get orders of magnitude improvements to storage space > and > > > > > > processing > > > > > > > >> speed. > > > > > > > >> > > > > > > > >> Let's first discuss the FINITE part. So far, the only > > arguments > > > I > > > > > have > > > > > > > read > > > > > > > >> for this are: > > > > > > > >> > > > > > > > >> 1. many other vector databases have it > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet better > > boast > > > > > having > > > > > > > >> "native" type support for vectors > > > > > > > >> 3. by focussing on this common special case, we can make > rapid > > > > > > progress, > > > > > > > >> while a general solution would take too long to specify > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will > be > > > in > > > > > the > > > > > > > >> vector > > > > > > > >> > > > > > > > >> the first two arguments are quite weak by themselves. Vector > > > > > databases > > > > > > > are > > > > > > > >> very specialized engines. Such engines can of course be very > > > > > > > opinionated on > > > > > > > >> their data types. A general format like Parquet must instead > > > weigh > > > > > > > whether > > > > > > > >> a special type is worth the added complexity over a more > > > general, > > > > > > widely > > > > > > > >> applicable type. In this case, it is IMHO not at all. And > just > > > > > having > > > > > > a > > > > > > > >> type called "vector" for advertisement reasons to make > Parquet > > > > > appear > > > > > > > more > > > > > > > >> modern seems a non-goal in the first place. > > > > > > > >> > > > > > > > >> The third "time to market" argument is valid for systems > that > > > > don't > > > > > > > need a > > > > > > > >> stable interface. It is a very dangerous attitude for > formats > > > like > > > > > > > Parquet > > > > > > > >> that must be stable long term and should have a > consolidated, > > > > > concise, > > > > > > > >> non-redundant type system. For such long term storage > formats, > > > we > > > > > need > > > > > > > to > > > > > > > >> take the time to build the right thing, as any shortcuts we > > take > > > > > will > > > > > > > stay > > > > > > > >> as blemishes in the format forever. If I read the discussion > > > > > > correctly, > > > > > > > >> this is actually the main argument as of now: We have found > > that > > > > it > > > > > is > > > > > > > hard > > > > > > > >> to align on how a fixed size list type should look like, so > we > > > > want > > > > > to > > > > > > > go > > > > > > > >> the easier route of a specialized type. This seems like a > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask > > yourself: > > > > > Would > > > > > > > we > > > > > > > >> want this, if Parquet already had a fixed size list type? > > > > > > > >> > > > > > > > >> So finally, the only technical argument is the fourth one: > "It > > > > > signals > > > > > > > >> semantics and ensures that no bad data lands in the > format". I > > > see > > > > > two > > > > > > > big > > > > > > > >> problems with this: > > > > > > > >> > > > > > > > >> 1. Readers *still* need to test for NaN / INF because > Parquet > > > > files > > > > > > may > > > > > > > be > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer from > > > > > physically > > > > > > > >> writing NaN or INF. *So you save no single line of code in a > > > > reader. > > > > > > You > > > > > > > >> save no branches, as you still need to handle the INF/NaN > > case.* > > > > So > > > > > > this > > > > > > > >> doesn't help with performance at all. The only difference > is, > > > that > > > > > the > > > > > > > >> error message changes from "found an invalid INF value in > the > > > > data" > > > > > to > > > > > > > >> "Your parquet file seems to be corrupt". It is actually > quite > > > the > > > > > > > opposite, > > > > > > > >> readers and writers get more complex having to handle this > new > > > > type. > > > > > > > >> > > > > > > > >> If you now argue that it's not about performance, it's about > > > > defined > > > > > > > >> semantics and erroring out early in writers, then the second > > > > problem > > > > > > > >> arises: > > > > > > > >> > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of > > > multiple > > > > > > > >> guarantees that you need for sensible vector processing. If > > "no > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would > agree > > > > that > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the only > > > guarantee > > > > > you > > > > > > > need, > > > > > > > >> instead, depending on the distance metric, you also need the > > > > > following > > > > > > > >> guarantees. E.g.: > > > > > > > >> > > > > > > > >> - No vector may be the 0 vector, otherwise, e.g., cosine > > > > > similarity > > > > > > is > > > > > > > >> undefined. Cosine similarity is often used as default, so > > this > > > > is > > > > > > > >> actually > > > > > > > >> quite important in practice. > > > > > > > >> - If we want to use a dot product, then all vectors need > to > > > > > > guarantee > > > > > > > a > > > > > > > >> length of 1. This would actually be quite useful for > > > performance > > > > > > > >> reasons. > > > > > > > >> - The vectors must come from the same model as the vector > > used > > > > for > > > > > > the > > > > > > > >> search. Otherwise, the search yields a numeric result, but > > > that > > > > > > result > > > > > > > >> has > > > > > > > >> no sensible meaning. > > > > > > > >> > > > > > > > >> So, what use is a type that just guarantees part of the > things > > > you > > > > > > need > > > > > > > for > > > > > > > >> vector search but not others? You still need additional > > > guarantees > > > > > in > > > > > > > >> addition to that type. What makes INF/NaN so important in > > > > comparison > > > > > > to, > > > > > > > >> say, "no 0 vector present", that it warrants an own type, > > while > > > > the > > > > > > > other > > > > > > > >> guarantees do not? If we really want this type as a basis > for > > > > > > > cross-engine > > > > > > > >> sharing of vectors, then it needs to give *all* guarantees. > > > > > > > >> > > > > > > > >> I'm not against a specialized vector type if it really > carries > > > > *all* > > > > > > > >> information to be useful by itself. If it carried all that > > > > > > information, > > > > > > > >> then a reading engine could indeed validate that only > > meaningful > > > > > > vector > > > > > > > >> searches are executed. But as long as it is just a > half-baked > > > > thing, > > > > > > it > > > > > > > is > > > > > > > >> not useful by itself and just a burden on each implementer > and > > > > > > > maintainer. > > > > > > > >> If we need extra information anyway, why not just stick to a > > > > general > > > > > > > type > > > > > > > >> like VECTOR (without FINITE). > > > > > > > >> > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or > tell > > > us) > > > > > what > > > > > > > >> advantage you will *really* get over just a (potentially > fixed > > > > size) > > > > > > > List > > > > > > > >> of Float? As discussed, it's not performance. It's also not > a > > > > > semantic > > > > > > > >> marker that is sufficient by itself. It's just a small hint > > that > > > > > > allows > > > > > > > >> writers to error out once they see a NaN or INF in the > input, > > > > while > > > > > > they > > > > > > > >> will still happily write a bogus 0 vector. This seems to be > of > > > > zero > > > > > > > >> benefit. > > > > > > > >> > > > > > > > >> Anecdotally, we currently handle petabytes of vectors in > > > > production > > > > > in > > > > > > > >> Data360. We simply store them in a List of Float, with an > > > > optimized > > > > > > > R-Level > > > > > > > >> reader implementation that rapidly checks that all vectors > are > > > the > > > > > > same > > > > > > > >> length. This solution works just fine. The overhead of > R-level > > > > > > decoding > > > > > > > is > > > > > > > >> barely measurable. The check is super fast and R-levels are > > tiny > > > > in > > > > > > > >> comparison to the payload vectors, so also the added I/O is > > > > > > negligible. > > > > > > > >> This is also why I'm so firm in my sentiment: We are > running a > > > > > > > production > > > > > > > >> grade system processing trillions of vectors right now and > see > > > no > > > > > big > > > > > > > >> issues with the types that Parquet currently provides. > > > > > > > >> > > > > > > > >> As I don't just want to be a naysayer and point out problems > > > with > > > > > the > > > > > > > >> current design, here are the directions that I would propose > > > > indeed: > > > > > > > >> > > > > > > > >> - a general fixed size list type. Don't do shortcuts just > > > > because > > > > > > the > > > > > > > >> discussion is too tedious. > > > > > > > >> - a vector type that indeed carries enough information to > be > > > > > really > > > > > > > >> useful for vector search by itself. (e.g., model name / > > > version, > > > > > > > >> optional > > > > > > > >> guarantees for no 0 vector / normalized vector length). > With > > > > such > > > > > a > > > > > > > >> type, > > > > > > > >> you could really do well defined vector searches even if > the > > > > > writing > > > > > > > >> engine > > > > > > > >> is completely different to the reading one. > > > > > > > >> - Standardize quantizations. This will be a real game > > changer > > > > for > > > > > > > >> performance. > > > > > > > >> > > > > > > > >> Cheers, > > > > > > > >> Jan > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu < > > > > > [email protected] > > > > > > >: > > > > > > > >>> > > > > > > > >>> I think separating them into FINITE_VECTOR and > > > FIXED_LENGTH_LIST > > > > > is a > > > > > > > >> good > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks > > > more > > > > > > > natural. > > > > > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR > > > first? > > > > > > > >>> > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev < > > > > > > > >>> [email protected]> wrote: > > > > > > > >>> > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to keep > > the > > > > name > > > > > > the > > > > > > > >>> same > > > > > > > >>>> as other engines/type systems. If we want a non-finite one > > in > > > > the > > > > > > > >> future > > > > > > > >>> we > > > > > > > >>>> can name that something else or we add fixed size list > that > > > will > > > > > > serve > > > > > > > >>> the > > > > > > > >>>> same purpose without a special name. > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue < > [email protected]> > > > > > wrote: > > > > > > > >>>> > > > > > > > >>>>> A logical type would: > > > > > > > >>>>> - Describe data for users and readers to understand what > > the > > > > file > > > > > > > >>>> contains > > > > > > > >>>>> - Signal that repetition and definition levels will > > reflect a > > > > > fixed > > > > > > > >>>> number > > > > > > > >>>>> of elements, which implementations can use to optimize > > > decoding > > > > > > > >>>>> - Guarantee that there are no null elements (because the > > type > > > > > > cannot > > > > > > > >> be > > > > > > > >>>>> applied to a list with optional elements) > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or > > > > Infinity > > > > > > > >> values > > > > > > > >>>> so > > > > > > > >>>>> that the vectors have a finite length and are suitable > for > > > > > distance > > > > > > > >>>>> calculations > > > > > > > >>>>> > > > > > > > >>>>> The finite value restriction is a restriction on writers > to > > > > > prevent > > > > > > > >> bad > > > > > > > >>>>> data from being written. > > > > > > > >>>>> > > > > > > > >>>>> Ignoring logical types that are not understood is a rule > > for > > > > > > > >> readers. A > > > > > > > >>>>> reader can handle vectors and pass it on to another > layer. > > > For > > > > > > > >>> instance, > > > > > > > >>>> a > > > > > > > >>>>> Parquet reader that doesn't support geometry passes > values > > as > > > > > > > >> `binary`. > > > > > > > >>>>> That binary value can be passed to a geometry UDF, even > if > > > the > > > > > > > >>> underlying > > > > > > > >>>>> engine and Parquet reader doesn't have support. > > > > > > > >>>>> > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can > similarly > > > > pass a > > > > > > > >> list > > > > > > > >>> of > > > > > > > >>>>> floats along to a UDF. But it is still important to the > > user > > > > that > > > > > > bad > > > > > > > >>>>> values are not present in the data. In your example, bad > > > values > > > > > are > > > > > > > >>>> vectors > > > > > > > >>>>> that cause the centroid calculation to fail, like vectors > > > with > > > > > > > >> infinite > > > > > > > >>>>> length. > > > > > > > >>>>> > > > > > > > >>>>> Ryan > > > > > > > >>>>> > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev < > > > > > > > >>>>> [email protected]> > > > > > > > >>>>> wrote: > > > > > > > >>>>> > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR annotation > on > > > > > > > >>> LIST<FLOAT>? > > > > > > > >>>>>> Specifically, what does it make possible that's > impossible > > > > > today, > > > > > > > >> and > > > > > > > >>>>> what > > > > > > > >>>>>> does it make easier that's currently hard? > > > > > > > >>>>>> > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like every > > > engine > > > > to > > > > > > > >>> offer > > > > > > > >>>>>> vector functions on lists of floats without needing a > new > > > > > logical > > > > > > > >>> type. > > > > > > > >>>>>> > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector type > > that > > > > > > > >>> describes > > > > > > > >>>>> how > > > > > > > >>>>>> to unpack and scale composite representations (quantised > > > codes > > > > > > plus > > > > > > > >>>>> scales, > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going. That > adds > > > > > > > >> information > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: > > engines > > > > > > already > > > > > > > >>>>>> understand the physical type, and many already offer > > vector > > > > > > > >> functions > > > > > > > >>>> on > > > > > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf > > > checks > > > > > are > > > > > > > >>>> cheap > > > > > > > >>>>>> enough that engines should do them anyway. The label > also > > > > > doesn't > > > > > > > >> say > > > > > > > >>>>> what > > > > > > > >>>>>> you can do with the column. Knowing that a column holds > > > > > embeddings > > > > > > > >>>>> doesn't > > > > > > > >>>>>> say how they can be used, just as zipcode and age are > both > > > > > > integers > > > > > > > >>> but > > > > > > > >>>>>> multiplying them is meaningless. > > > > > > > >>>>>> > > > > > > > >>>>>> My concern is the opposite effect. We're establishing > that > > > > > engines > > > > > > > >>>> should > > > > > > > >>>>>> ignore logical types they don't understand. So a tool > that > > > > > doesn't > > > > > > > >>> know > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user > > > > compute > > > > > > > >>>>> centroids > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other > > > engines > > > > > > > >>> restrict > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns, that > file > > > > stops > > > > > > > >>>> working > > > > > > > >>>>>> with them, through no fault of the user. An annotation > > that > > > > > > doesn't > > > > > > > >>>>> change > > > > > > > >>>>>> what the data is shouldn't be able to make it unusable. > > > > > > > >>>>>> > > > > > > > >>>>>> Am I missing a use case? > > > > > > > >>>>>> > > > > > > > >>>>>> Best, Will > > > > > > > >>>>>> > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu < > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >>>>>> > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of > values > > to > > > > > many > > > > > > > >>> use > > > > > > > >>>>>> cases. > > > > > > > >>>>>>> It is would be great to make progress on the > > FINITE_VECTOR, > > > > and > > > > > > > >> not > > > > > > > >>>> to > > > > > > > >>>>>> fit > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST. > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue < > > [email protected]> > > > > > > > >> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>>> Going through this thread again, it looks like there > > are a > > > > few > > > > > > > >>>> things > > > > > > > >>>>>>> that > > > > > > > >>>>>>>> people generally agree on: > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> - In the short term, we want to introduce a logical > type > > > or > > > > > > > >>> types; > > > > > > > >>>> a > > > > > > > >>>>>>> better > > > > > > > >>>>>>>> physical representation can be done later. We can > focus > > on > > > > > > > >>> logical > > > > > > > >>>>>> types. > > > > > > > >>>>>>>> - There is precedent in logical types for restricting > > > > values, > > > > > > > >>> like > > > > > > > >>>>> UTF8 > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we > > also > > > > > agree > > > > > > > >>>> that > > > > > > > >>>>> we > > > > > > > >>>>>>>> could restrict vector elements to non-nested or > > primitive > > > > > > > >> types. > > > > > > > >>>>>>>> - There are many examples of databases that restrict > > their > > > > > > > >> vector > > > > > > > >>>>> types > > > > > > > >>>>>>> to > > > > > > > >>>>>>>> non-null, finite elements. > > > > > > > >>>>>>>> - Some people want to have write-time guarantees that > > > > vectors > > > > > > > >>> have > > > > > > > >>>>> only > > > > > > > >>>>>>>> non-null, finite elements > > > > > > > >>>>>>>> - Some people want to extend this work to include > > tensors, > > > > > > > >>> shorter > > > > > > > >>>>>>> floating > > > > > > > >>>>>>>> point representations, variable lengths, NaN or > Infinity > > > > > > > >>> elements, > > > > > > > >>>>> etc. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two > halves > > of > > > > the > > > > > > > >>>>>> community > > > > > > > >>>>>>>> that care about different things. That just means we > > > should > > > > > > > >> have > > > > > > > >>>>>> multiple > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type that > > allows > > > > NaN > > > > > > > >>>>> values, > > > > > > > >>>>>>> and > > > > > > > >>>>>>>> it's also reasonable for those of us building > databases, > > > > table > > > > > > > >>>>>> standards, > > > > > > > >>>>>>>> and vector indexes to care about guarantees that > > elements > > > > are > > > > > > > >>>> finite. > > > > > > > >>>>>>>> There's more than enough examples to show that this > is a > > > > > > > >>> legitimate > > > > > > > >>>>> use > > > > > > > >>>>>>>> case. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Since I don't see the concerns of the different use > > cases > > > > > > > >>> changing, > > > > > > > >>>>> it > > > > > > > >>>>>>>> looks like we need to find a compromise to move > > forward. I > > > > > > > >>> propose > > > > > > > >>>>> that > > > > > > > >>>>>>> we > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a > > > > > > > >> FIXED_LENGTH_LIST > > > > > > > >>>> or > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN > values. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason > > to > > > > try > > > > > > > >> to > > > > > > > >>>> fit > > > > > > > >>>>>>> these > > > > > > > >>>>>>>> use cases into a single type? What if that means not > > > making > > > > > > > >>>> progress? > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Ryan > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via > > dev > > > < > > > > > > > >>>>>>>> [email protected]> wrote: > > > > > > > >>>>>>>> > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with pulling > > out > > > > the > > > > > > > >>>>>> “NaN/Inf” > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my > > preference), > > > or > > > > > > > >>>> putting > > > > > > > >>>>>> no > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE > > 754 > > > > > > > >> float > > > > > > > >>>>>> numbers. > > > > > > > >>>>>>>> If > > > > > > > >>>>>>>>> some top-level application wants to put additional > > > > > > > >> constraints > > > > > > > >>> on > > > > > > > >>>>> the > > > > > > > >>>>>>>> data > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can > > > still > > > > do > > > > > > > >>> so > > > > > > > >>>>> one > > > > > > > >>>>>>>> layer > > > > > > > >>>>>>>>> above. It seems like this was the biggest point of > > > > discussion > > > > > > > >>>>>>> remaining. > > > > > > > >>>>>>>>> All these options work for me, and we should move > > forward > > > > > > > >> with > > > > > > > >>>> this > > > > > > > >>>>>>>>> proposal. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Concerning additional element types like bf16, they > can > > > be > > > > > > > >>>>> introduced > > > > > > > >>>>>>> as > > > > > > > >>>>>>>>> Parquet types (at least in the current LIST-based > spec > > > > > > > >>> proposal). > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Concerning Will’s suggestion around representing > VECTOR > > > as > > > > a > > > > > > > >>>>> variable > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata > > restriction: I > > > > > > > >> think > > > > > > > >>>>> that > > > > > > > >>>>>> a > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides enough > > > > benefits > > > > > > > >> by > > > > > > > >>>>>>>> describing > > > > > > > >>>>>>>>> the semantics of the data, and allowing to capture > > future > > > > > > > >>> changes > > > > > > > >>>>>>> around > > > > > > > >>>>>>>>> statistics. In that case, the length is better > captured > > > as > > > > an > > > > > > > >>>> exact > > > > > > > >>>>>>>> length > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure about > > other > > > > use > > > > > > > >>>> cases > > > > > > > >>>>>> for > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint > system > > > > just > > > > > > > >>> for > > > > > > > >>>>> the > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have > > > enough > > > > > > > >>> value > > > > > > > >>>> in > > > > > > > >>>>>> my > > > > > > > >>>>>>>>> view. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Best, > > > > > > > >>>>>>>>> Philipp > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > > > > > > > >>>>>>>>> [email protected]> > > > > > > > >>>>>>>>> wrote: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond > > > > > > > >> normalized > > > > > > > >>>>>>> embeddings > > > > > > > >>>>>>>>> and > > > > > > > >>>>>>>>>> engines really aren't going to be able to elide all > > the > > > > > > > >>> checks > > > > > > > >>>>>>> anyway. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions > > too > > > > :) > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) > > can't > > > > > > > >>>> express > > > > > > > >>>>>> this > > > > > > > >>>>>>>> in > > > > > > > >>>>>>>>>> Parquet. Engines can write opaque blobs they > > understand > > > > in > > > > > > > >>> the > > > > > > > >>>>>>>> metadata > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info > round-trips, > > > but > > > > > > > >>> this > > > > > > > >>>>>> isn't > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new > > > Thrift > > > > > > > >>>> fields > > > > > > > >>>>>> (my > > > > > > > >>>>>>>>> pref) > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being > > more > > > > > > > >>>> broadly > > > > > > > >>>>>>> useful > > > > > > > >>>>>>>>> for > > > > > > > >>>>>>>>>> other use cases and data types? > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the > > MIN_LENGTH > > > > and > > > > > > > >>>>>>> MAX_LENGTH > > > > > > > >>>>>>>>> for > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types. These > would > > > > > > > >> count > > > > > > > >>>>>> logical > > > > > > > >>>>>>>>> units > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than > bytes. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to > Parquet > > as > > > > > > > >>>> LIST(N) > > > > > > > >>>>>> with > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even > > > > > > > >> different > > > > > > > >>>>>>> engines - > > > > > > > >>>>>>>>> can > > > > > > > >>>>>>>>>> pick up these constraints if they support doing so. > > We > > > > > > > >> don't > > > > > > > >>>>>>> actually > > > > > > > >>>>>>>>> need > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and > VARCHAR(N) > > > to > > > > > > > >> be > > > > > > > >>>>>>> expressed > > > > > > > >>>>>>>> on > > > > > > > >>>>>>>>>> UTF8 columns etc. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> This also seems generally useful in the future as we > > > > > > > >> approach > > > > > > > >>>>>>>> expressing > > > > > > > >>>>>>>>>> other tensor data shapes and types. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage type > > > doesn't > > > > > > > >>>> enforce > > > > > > > >>>>>>> these > > > > > > > >>>>>>>>>> constraints and engines must check them. It's > > previously > > > > > > > >>>>>> established > > > > > > > >>>>>>>> that > > > > > > > >>>>>>>>>> performance is just a software problem and there is > no > > > > > > > >>>>> fundamental > > > > > > > >>>>>>>>>> performance advantage to the fixed size types in > > > > > > > >> Dremel-like > > > > > > > >>>>>> formats > > > > > > > >>>>>>>> like > > > > > > > >>>>>>>>>> Parquet. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Best, > > > > > > > >>>>>>>>>> Will > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard > < > > > > > > > >>>>>>>>> [email protected] > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> wrote: > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I > > > have > > > > > > > >>> been > > > > > > > >>>>>>>> thinking > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would share > my > > 2 > > > > > > > >>> cents. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be > finite, > > > as > > > > > > > >>>>> proposed > > > > > > > >>>>>>> in > > > > > > > >>>>>>>>> the > > > > > > > >>>>>>>>>> PR > > > > > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624 > >. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> While Divjot and others note the specialization for > > > > > > > >> UTF-8, > > > > > > > >>>>>> Parquet > > > > > > > >>>>>>>>>> already > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings > > > > > > > >>>>>>>>>>> < > > > > > > > >>>>>>>> > > > > > > > >>>> > > > > > > > > https://github.com/apache/parquet-format/blob/master/README.md#types > > > > > > > >>>>>>>>>> . > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would > be a > > > new > > > > > > > >>>>>>> restriction > > > > > > > >>>>>>>>> on > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar > > > FLOAT/DOUBLE > > > > > > > >>>>> columns > > > > > > > >>>>>>>> have > > > > > > > >>>>>>>>>>> never had such a rule. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> It would also not match how the systems that > actually > > > > > > > >>> produce > > > > > > > >>>>>> these > > > > > > > >>>>>>>>>> vectors > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. > > Spark, > > > > > > > >>>> DuckDB, > > > > > > > >>>>>>>> NumPy, > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, > and > > > > > > > >>>> OpenBLAS. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an > > > > > > > >> insert > > > > > > > >>>>>> policy. > > > > > > > >>>>>>>>> While > > > > > > > >>>>>>>>>> I > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format should > > allow > > > > > > > >> them > > > > > > > >>>> out > > > > > > > >>>>>> of > > > > > > > >>>>>>>> the > > > > > > > >>>>>>>>>> box. > > > > > > > >>>>>>>>>>> It is a user system decision to disallow writing > > these > > > > > > > >>> values > > > > > > > >>>>> or > > > > > > > >>>>>> to > > > > > > > >>>>>>>>> throw > > > > > > > >>>>>>>>>>> errors when encountering them. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of > > > vector > > > > > > > >>>> math. > > > > > > > >>>>>>>> Finite > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance: > magnitude = > > > > > > > >>>>>> ||[FLOAT_MAX, > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite > > overflows > > > > > > > >> to > > > > > > > >>>>> +Inf. > > > > > > > >>>>>>>>> PyTorch > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20, > > > 1e20]).norm() > > > > > > > >> is > > > > > > > >>>> inf > > > > > > > >>>>>> in > > > > > > > >>>>>>>>> fp32. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf > > > while > > > > > > > >>> the > > > > > > > >>>>>>> VECTOR > > > > > > > >>>>>>>>>> itself > > > > > > > >>>>>>>>>>> was required to be finite. The ban only forbids > > storing > > > > > > > >>>>> specials > > > > > > > >>>>>> in > > > > > > > >>>>>>>> the > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple > > > downstream > > > > > > > >>>>>>> computations > > > > > > > >>>>>>>>>> from > > > > > > > >>>>>>>>>>> producing them. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an > > > optional > > > > > > > >>> bool > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that > want > > > > > > > >>>>>> pgvector-like > > > > > > > >>>>>>>>>>> rejection can require the flag = true. Engines that > > use > > > > > > > >>>> NumPy / > > > > > > > >>>>>>>>> PyTorch / > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. > For > > > the > > > > > > > >>> same > > > > > > > >>>>>>>> reason, I > > > > > > > >>>>>>>>>> do > > > > > > > >>>>>>>>>>> not favor adding yet another type called > > FINITE_VECTOR. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> Best, > > > > > > > >>>>>>>>>>> Sebastian > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via > dev > > < > > > > > > > >>>>>>>>>>> [email protected]> > > > > > > > >>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>>> I agree with moving forward with the proposal and > > > > > > > >>> renaming > > > > > > > >>>>>> VECTOR > > > > > > > >>>>>>>> to > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something > > > > > > > >>>>>>>>>>>> similar if others feel that better communicates > its > > > > > > > >>>> intent. I > > > > > > > >>>>>> see > > > > > > > >>>>>>>>> value > > > > > > > >>>>>>>>>>> in > > > > > > > >>>>>>>>>>>> having a vector type that guarantees > > > > > > > >>>>>>>>>>>> its values are mathematically-correct vectors for > a > > > few > > > > > > > >>>>>> reasons: > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to > the > > > > > > > >>> STRING > > > > > > > >>>>>>> logical > > > > > > > >>>>>>>>> type > > > > > > > >>>>>>>>>>>> mandating UTF-8. > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for > > > > > > > >> detecting > > > > > > > >>>> NaN > > > > > > > >>>>>> or > > > > > > > >>>>>>>>> +-inf > > > > > > > >>>>>>>>>>>> values as all statistics are optional. > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for > requiring > > > > > > > >> that > > > > > > > >>>>>> writers > > > > > > > >>>>>>>>> emit > > > > > > > >>>>>>>>>>>> stats for specific logical types. > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow > > > > > > > >> through > > > > > > > >>>> the > > > > > > > >>>>>>> engine > > > > > > > >>>>>>>>> and > > > > > > > >>>>>>>>>>>> block them for specific functionality. > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I > don't > > > > > > > >> think > > > > > > > >>>>> this > > > > > > > >>>>>> is > > > > > > > >>>>>>>> as > > > > > > > >>>>>>>>>> easy > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for > > examples). > > > > > > > >>> Their > > > > > > > >>>>>>> Python > > > > > > > >>>>>>>>> [3] > > > > > > > >>>>>>>>>>> and > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on > > > > > > > >>>>>>>>>>>> vectors that contain NaN values. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, > > but > > > I > > > > > > > >>>> don't > > > > > > > >>>>>>> think > > > > > > > >>>>>>>>>> this > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types. > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet > > > > > > > >> would > > > > > > > >>> be > > > > > > > >>>>>>>>> considered > > > > > > > >>>>>>>>>> a > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list type > if > > > we > > > > > > > >>>>> believe > > > > > > > >>>>>>>> there > > > > > > > >>>>>>>>>> are > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> [1] > https://github.com/lance-format/lance/pull/3648 > > > > > > > >>>>>>>>>>>> [2] > > https://github.com/lance-format/lance/issues/8607 > > > > > > > >>> (see > > > > > > > >>>>>>> point 2 > > > > > > > >>>>>>>>> in > > > > > > > >>>>>>>>>>>> "root > > > > > > > >>>>>>>>>>>> cause") > > > > > > > >>>>>>>>>>>> [3] > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > > > >>>>>>>>>>>> [4] > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck > > via > > > > > > > >>> dev < > > > > > > > >>>>>>>>>>>> [email protected]> wrote: > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as > > is. > > > > > > > >> I > > > > > > > >>>> see > > > > > > > >>>>>>> strong > > > > > > > >>>>>>>>>> value > > > > > > > >>>>>>>>>>>> and > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I > agree > > > > > > > >> with > > > > > > > >>>> the > > > > > > > >>>>>>>>>>> restrictions > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in > general. > > > > > > > >>>>> Precedent > > > > > > > >>>>>>>>> exists > > > > > > > >>>>>>>>>> in > > > > > > > >>>>>>>>>>>>> other types that restrict which physical type > > values > > > > > > > >>> are > > > > > > > >>>>>>>> logically > > > > > > > >>>>>>>>>>> valid > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be > > > > > > > >> WKB-encoded). > > > > > > > >>>> A > > > > > > > >>>>>>>> general > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it > > makes > > > > > > > >>>> sense > > > > > > > >>>>> to > > > > > > > >>>>>>>>>> separate > > > > > > > >>>>>>>>>>>> the > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can > > see > > > > > > > >>> how > > > > > > > >>>>> the > > > > > > > >>>>>>> two > > > > > > > >>>>>>>>>> types > > > > > > > >>>>>>>>>>>>> might diverge further in the future regarding > > > > > > > >>> statistics > > > > > > > >>>> or > > > > > > > >>>>>>>>>> encodings. > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector type, > but > > it > > > > > > > >>>> seems > > > > > > > >>>>>>> like > > > > > > > >>>>>>>>> the > > > > > > > >>>>>>>>>>>> VECTOR > > > > > > > >>>>>>>>>>>>> name is already used across other engines > mentioned > > > > > > > >> in > > > > > > > >>>> this > > > > > > > >>>>>>>> thread. > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> Best, > > > > > > > >>>>>>>>>>>>> Philipp > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < > > > > > > > >>>> [email protected]> > > > > > > > >>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we > should > > > > > > > >>> treat > > > > > > > >>>>> the > > > > > > > >>>>>>>>> general > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout, > > > > > > > >> and > > > > > > > >>>>>> separate > > > > > > > >>>>>>>> it > > > > > > > >>>>>>>>>> from > > > > > > > >>>>>>>>>>>> the > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader > ML > > > > > > > >>>> tensor) > > > > > > > >>>>>>>> logical > > > > > > > >>>>>>>>>>> type. > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a > > > > > > > >>>> specialized > > > > > > > >>>>>>> VECTOR > > > > > > > >>>>>>>>>> type > > > > > > > >>>>>>>>>>> so > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a common > > > > > > > >>>> semantic > > > > > > > >>>>>>>>> contract > > > > > > > >>>>>>>>>>> for > > > > > > > >>>>>>>>>>>>>> interoperability. > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just the > > > > > > > >>>> NaN/Inf/null > > > > > > > >>>>>>>>>> limitations, > > > > > > > >>>>>>>>>>>>> maybe > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to > > > > > > > >> the > > > > > > > >>>> name > > > > > > > >>>>>>> (like > > > > > > > >>>>>>>>>>> Andrew > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or > > FINITE_VECTOR) > > > > > > > >>> to > > > > > > > >>>>>> avoid > > > > > > > >>>>>>>>>>> confusion > > > > > > > >>>>>>>>>>>>> with > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could > > > > > > > >>>> introduce a > > > > > > > >>>>>>>> boolean > > > > > > > >>>>>>>>>>> flag > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as > > proposed > > > > > > > >>>>>> earlier. > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > > > > > > >>>>>>>>>> [email protected]> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points of > > > > > > > >>>> disagreement > > > > > > > >>>>>>> here. > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for > > > > > > > >> the > > > > > > > >>>>>>> specific > > > > > > > >>>>>>>>> use > > > > > > > >>>>>>>>>>> case > > > > > > > >>>>>>>>>>>>> of > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used > > > > > > > >> for > > > > > > > >>>>>> vector > > > > > > > >>>>>>>>>>>> embeddings? > > > > > > > >>>>>>>>>>>>>> If > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the > > > > > > > >> name > > > > > > > >>>>>>>>>> (FINITE_VECTOR? > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining > > > > > > > >>> logical > > > > > > > >>>>>> types > > > > > > > >>>>>>>>> that > > > > > > > >>>>>>>>>>>>> specify > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their > > > > > > > >> contents > > > > > > > >>> to > > > > > > > >>>>>> help > > > > > > > >>>>>>>>>> readers > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or > JSON; > > > > > > > >>>>>> Defining > > > > > > > >>>>>>>> one > > > > > > > >>>>>>>>>> for > > > > > > > >>>>>>>>>>>>>> vector > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> Andrew > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> [1]: > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > > > > > > > >>>>>>>>> [email protected]> > > > > > > > >>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits > > > > > > > >>>> their > > > > > > > >>>>>>>> usage, > > > > > > > >>>>>>>>>> but > > > > > > > >>>>>>>>>>>>> we're > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format > > > > > > > >>> that > > > > > > > >>>>> aims > > > > > > > >>>>>>> to > > > > > > > >>>>>>>> be > > > > > > > >>>>>>>>>>>> broadly > > > > > > > >>>>>>>>>>>>>>>>> applicable. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be > > > > > > > >> general > > > > > > > >>>>>>> purpose, > > > > > > > >>>>>>>>> but > > > > > > > >>>>>>>>>>>> types > > > > > > > >>>>>>>>>>>>>>> within > > > > > > > >>>>>>>>>>>>>>>> the format define specific representation and > > > > > > > >>>>> semantics > > > > > > > >>>>>>> to > > > > > > > >>>>>>>>>>> achieve > > > > > > > >>>>>>>>>>>>> that > > > > > > > >>>>>>>>>>>>>>>> goal. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > > > > > > > >>>> it's > > > > > > > >>>>>>> quite > > > > > > > >>>>>>>>>>> trivial > > > > > > > >>>>>>>>>>>> to > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > > > > > > > >>> e.g. > > > > > > > >>>>>> NumPy > > > > > > > >>>>>>>> or > > > > > > > >>>>>>>>>>> Pandas > > > > > > > >>>>>>>>>>>>> (or > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > > > > > > > >>>>> computing > > > > > > > >>>>>>>>>> software). > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite > > > > > > > >>>> values > > > > > > > >>>>> is > > > > > > > >>>>>>>>>> directly > > > > > > > >>>>>>>>>>>>>> related > > > > > > > >>>>>>>>>>>>>>> to > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector. Vectors > have > > > > > > > >>>>>> specific > > > > > > > >>>>>>>>>>>>>> characteristics > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). > > > > > > > >>> Non-finite > > > > > > > >>>>>>> values > > > > > > > >>>>>>>>> are > > > > > > > >>>>>>>>>>>>> invalid > > > > > > > >>>>>>>>>>>>>>> and > > > > > > > >>>>>>>>>>>>>>>> break those characteristics. I'd liken it to > > > > > > > >>>> storing > > > > > > > >>>>>>>>> non-utf8 > > > > > > > >>>>>>>>>>>> values > > > > > > > >>>>>>>>>>>>>> in > > > > > > > >>>>>>>>>>>>>>> a > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed > > > > > > > >>>> length > > > > > > > >>>>>>> list, > > > > > > > >>>>>>>>>> which > > > > > > > >>>>>>>>>>>> is > > > > > > > >>>>>>>>>>>>>> more > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic > > > > > > > >>>> meaning. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all > > > > > > > >>> similarly > > > > > > > >>>>>> shaped > > > > > > > >>>>>>>> use > > > > > > > >>>>>>>>>>> cases > > > > > > > >>>>>>>>>>>>>> into > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list). > > > > > > > >> Vector > > > > > > > >>>> is > > > > > > > >>>>> a > > > > > > > >>>>>>> well > > > > > > > >>>>>>>>>>>>> established > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the > > > > > > > >>> general > > > > > > > >>>>>>> purpose > > > > > > > >>>>>>>>>>> nature > > > > > > > >>>>>>>>>>>> of > > > > > > > >>>>>>>>>>>>>> the > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases > > > > > > > >> natively > > > > > > > >>>>>> support > > > > > > > >>>>>>>>>> this). > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> -Dan > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine > > > > > > > >> Pitrou < > > > > > > > >>>>>>>>>>>> [email protected]> > > > > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a > > > > > > > >> écrit > > > > > > > >>> : > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the > > > > > > > >>>> clear > > > > > > > >>>>>>>>> use-case > > > > > > > >>>>>>>>>>> for > > > > > > > >>>>>>>>>>>>>>>> non-finite > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has > > > > > > > >> been a > > > > > > > >>>> lot > > > > > > > >>>>>> of > > > > > > > >>>>>>>>>>>> theoretical > > > > > > > >>>>>>>>>>>>>>>>> discussion > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings > > > > > > > >>>> currently > > > > > > > >>>>>>>> produce > > > > > > > >>>>>>>>>>> these > > > > > > > >>>>>>>>>>>>>>> vectors > > > > > > > >>>>>>>>>>>>>>>>> and > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines? > > > > > > > >>> I'd > > > > > > > >>>>>> like > > > > > > > >>>>>>> to > > > > > > > >>>>>>>>>> limit > > > > > > > >>>>>>>>>>>>>>>> introducing > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values > > > > > > > >> which > > > > > > > >>>> are > > > > > > > >>>>>>> simply > > > > > > > >>>>>>>>> are > > > > > > > >>>>>>>>>>>>> failure > > > > > > > >>>>>>>>>>>>>>>> modes > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and > > > > > > > >>>> it's > > > > > > > >>>>>>> quite > > > > > > > >>>>>>>>>>> trivial > > > > > > > >>>>>>>>>>>> to > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using > > > > > > > >>> e.g. > > > > > > > >>>>>> NumPy > > > > > > > >>>>>>>> or > > > > > > > >>>>>>>>>>> Pandas > > > > > > > >>>>>>>>>>>>> (or > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific > > > > > > > >>>>> computing > > > > > > > >>>>>>>>>> software). > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector > > > > > > > >>> is > > > > > > > >>>> a > > > > > > > >>>>>> IETF > > > > > > > >>>>>>>>>>> standard > > > > > > > >>>>>>>>>>>> or > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8 > > > > > > > >> (where > > > > > > > >>>> the > > > > > > > >>>>>>>> values > > > > > > > >>>>>>>>>> are > > > > > > > >>>>>>>>>>>>>> expected > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective > > > > > > > >> specs) > > > > > > > >>> do > > > > > > > >>>>> not > > > > > > > >>>>>>>>> apply. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better > > > > > > > >> chance > > > > > > > >>>> of > > > > > > > >>>>>>>>> converging > > > > > > > >>>>>>>>>>> if > > > > > > > >>>>>>>>>>>> we > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of: > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data > > > > > > > >>> in > > > > > > > >>>>>>> Parquet > > > > > > > >>>>>>>>>> (with > > > > > > > >>>>>>>>>>> a > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types) > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given > > > > > > > >>>> column > > > > > > > >>>>>> (or > > > > > > > >>>>>>>>>> column > > > > > > > >>>>>>>>>>>>> chunk, > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own > > > > > > > >>> right, > > > > > > > >>>>> but > > > > > > > >>>>>> do > > > > > > > >>>>>>>> not > > > > > > > >>>>>>>>>>> have > > > > > > > >>>>>>>>>>>> to > > > > > > > >>>>>>>>>>>>>> be > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Regards > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Antoine. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
