Hi Ryan, Thanks for the detailed replies. Let me restate my position more directly, since I don't think I made it clear.
I'd like to put the vector type on hold in Parquet until there's a general proposal for ML tensors and sparse, packed or quantised representations. That's where a new type would let engines do interoperable things they can't do today. My earlier question was what VECTOR makes possible that isn't possible today. The answers so far describe what the annotation asserts: that the data is a vector, that it has a fixed length, and that its values are finite and non-null. I'm asking for a workload that can't be done on LIST<FLOAT> now and would work with VECTOR. Jan asked the same question, and I don't think either of us has seen one yet. For embeddings, LIST<FLOAT> is what is already widely used today, and it works. My compatibility concern is about writers. A tool that doesn't know VECTOR rewrites the file as plain LIST<FLOAT>. If engines then require the annotation, that data stops working. If they don't require it, the annotation adds little. A world where VECTOR exists and plain LIST<FLOAT> still works fully for ANN etc is a world that doesn't need VECTOR. On optimisations, Dan framed this as semantics rather than performance, but you've listed decoding optimisations. In my experience readers can remove the cost of variable-length lists without any format change, and I'm happy to share the recipes with anyone interested. If fixed length is still the real gain, a fixed-size list gives it to everyone, NaN or not. If table formats need a finite-vector contract, I think the table layer is a better home than the file format. An Iceberg type would apply equally to Parquet, Avro and ORC. I'd still be wary of Iceberg rejecting untagged columns, but it's a better place for that debate than Parquet. Best, Will On Thu, 22 Oct 2026 at 02:00, Ryan Blue <[email protected]> wrote: > Thanks, Curt. I wasn't aware of that behavior in MySQL. I was trying to > learn more and I don't see it in documentation either. Do you have any more > information about how MySQL handles this? One page makes it sound like it > may be platform dependent > <https://dev.mysql.com/doc/refman/26.7/en/problems-with-float.html>, where > some values produce "inf" and some produce "0". > > My original comment was referring to databases that allow users to specify > types that exclude non-finite values, which is what we would need in order > to support a finite float in table formats since we already support the > full range of values. If you know of any cases like that rather than just > limiting values, I'm curious. Thanks! > > Ryan > > On Wed, Oct 21, 2026 at 3:48 PM Curt Hagenlocher <[email protected]> > wrote: > > > > I'm not aware of databases with finite floating point types. > > > > Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow > > storing non-finite floating-point values in their binary > > floating-point columns. They don't have a separate type which does > > allow this. > > > > On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <[email protected]> wrote: > > > > > > Jan, > > > > > > I agree that we _could_ model vectors as an orthogonal finiteness > > property > > > for elements in a fixed-length list. I don't know how useful that would > > be. > > > I've not had anyone ask for that, and I'm not aware of databases with > > > finite floating point types. I think it makes more sense to move > forward > > > with the existing FINITE_VECTOR that is proposed. > > > > > > For the logical type argument, it sounds like the issue is that you > want > > to > > > fall back to readers treating vectors as LIST. I don't see a good way > to > > > make that happen, but it also isn't needed by the table format use case > > > because we are introducing this as a new type. Do you have a use case > > where > > > you are going to mix FINITE_VECTOR with LIST files and this will cause > a > > > problem? > > > > > > I don't think that we've had an expectation for logical types in the > past > > > that they produce a particular in-memory representation. Maybe we could > > > allow setting both `LIST` and `FINITE_VECTOR` logical types at the same > > > time? This kind of compatibility hasn't been a priority as far as I'm > > > aware, but if it is important to you we can consider it. > > > > > > Ryan > > > > > > On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <[email protected]> wrote: > > > > > > > assuming what will be proposed is a new logical type FINITE_VECTOR > > that > > > > annotates the same column that would have been annotated as LIST > > before: > > > > > > > > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness > is > > part > > > > of the vector type. Isn't the finiteness rather a property of the > > contained > > > > floats? Such a modeling would be more composable, as it would for > > example > > > > also allow to have normal (non-vector) float columns that are > > restricted to > > > > be finite. So, if we do think that we want finiteness in the type > > system, I > > > > would rather cut it into two parts: A fixed size vector concept, and > a > > new > > > > way of labeling float columns finite. The vector type would then just > > boil > > > > down to be a fixed size list / we could just give the list type an > > optional > > > > fixed size field. Then, the column contained within it would be a > float > > > > tagged as finite. > > > > > > > > It was mentioned that a logical type annotating a list of float would > > be > > > > forward compatible, but - depending on the modeling - that is not > > > > necessarily true: as logical types in Parquet do not support > > subtyping, a > > > > column cannot be a list of float and a (finite) vector at the same > > time, > > > > but only one of the two. It depends on how the type will be modeled > in > > > > thrift in the end, but if it is a new union member of `LogicalType`, > > then > > > > legacy engines will not understand that such a vector can also just > be > > > > interpreted as a list of floats. They will just see a nested column > > type of > > > > a logical type they don't understand, so they will likely reject > > reading > > > > this column (at least that's what the engine I'm maintaining would be > > > > doing). > > > > > > > > In contrast, if we just give the list type an optional fixed size and > > float > > > > an optional finiteness, then the resulting thrift is actually fully > > forward > > > > compatible: Legacy engines will just ignore the fixed length and > treat > > the > > > > list as variable length and they will ignore the finiteness and treat > > the > > > > element type as float. > > > > > > > > (A side note, which most people here may already be aware of, but > it's > > good > > > > to repeat it: The not-NULL property doesn't need new type support at > > all. > > > > The normal type system of list already allows setting the contained > > element > > > > type to be optional or required. If it is set to be required, then > > NULLs > > > > are not allowed.) > > > > > > > > Cheers, > > > > Jan > > > > > > > > > > > > > > > > > > > > > > > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer < > > > > [email protected]>: > > > > > > > > > I support proceeding with the two types. I honestly think the cost > of > > > > > adding new logical types is overestimated. If we have as many > > interested > > > > > parties as have appeared in this thread we already know there is a > > large > > > > > group of engineers ready to do the work and engines ready to > consume > > (and > > > > > produce) the type. The fact that implementations currently use > > > > heuristics, > > > > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC > > > > > Glasgow) or custom annotations to achieve the same performance > > benefits > > > > > makes me think we should just create the type and move on. > > > > > > > > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <[email protected]> wrote: > > > > > > > > > > > An annotation on list<float> is useful because: > > > > > > - We want to correctly annotate what is being stored > > > > > > - We want to unlock optimizations when each non-null list has the > > same > > > > > > number of elements > > > > > > - We want to have writer guarantees > > > > > > > > > > > > I don't think that logical type compatibility is a good argument. > > We > > > > want > > > > > > to avoid a case where we have to treat all binary like a string. > > And > > > > > > because we are introducing this in table formats after > > standardizing it > > > > > > here, we can do that reliably. > > > > > > > > > > > > As I said in my last email, I think there's a difference between > > > > > finiteness > > > > > > and general "invariants" you might want. Vectors aren't vectors > if > > they > > > > > > don't have a length and a direction. Other invariants don't work > > the > > > > same > > > > > > way. A zero-length vector is still a valid vector and can be > used, > > just > > > > > not > > > > > > for certain distance metrics. > > > > > > > > > > > > I also want to clarify that when we looked into this more closely > > and > > > > > > recognized the mismatch between the two separate use cases, I > > offered > > > > to > > > > > > split out the finite vector work so that we didn't cause any > > headaches > > > > > for > > > > > > people who are interested in fixed-length list. It sounds like > not > > > > > > splitting out the use cases is causing a lot of headache. Vectors > > for > > > > our > > > > > > use case need a finiteness guarantee at write time, and > > fixed-length > > > > > lists > > > > > > need to store NaN and other values. > > > > > > > > > > > > It's okay to have two types and we should for two different use > > cases. > > > > > > Otherwise, we are forcing one to make an unnecessary compromise. > I > > > > don't > > > > > > want to force everyone to exclude NaN and infinite values from > > > > > fixed-length > > > > > > lists. Seems reasonable that we should also be able to have > vectors > > > > that > > > > > > exclude those values, since that is how they are used and we have > > to > > > > also > > > > > > guarantee it for exchanging data with other existing databases. > > > > > > > > > > > > Ryan > > > > > > > > > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > It reads well. Three small fixes: > > > > > > > > > > > > > > 1. **"back in May or so"** is ambiguous. The > > > > > fixed-size-list-for-tensors > > > > > > > framing is from Rok's opening post in May 2024. The May 2026 > > design > > > > doc > > > > > > was > > > > > > > already mostly about performance and layout. "Originally" > avoids > > > > anyone > > > > > > > correcting the date. > > > > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't > > quite > > > > > > > right. The NaN-for-missing behaviour comes from > > `VectorAssembler` and > > > > > > > `Imputer` and applies to Spark ML vectors generally, dense or > > sparse. > > > > > > > "Spark's feature vectors, dense and sparse," fixes it. > > > > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark, > > BigQuery and > > > > > > > DuckDB and pretty much…" has a doubled "and". > > > > > > > > > > > > > > The last sentence of the invariants paragraph could be clearer. > > > > > "Although > > > > > > > that doesn't mean readers shouldn't check on read" has a double > > > > > negative > > > > > > > and is easy to misread. "Though careful readers will still > check > > on > > > > > read" > > > > > > > is simpler. > > > > > > > > > > > > > > With those applied: > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > The scope of this discussion keeps moving :). IIRC it > originally > > was > > > > > > about > > > > > > > fixed-size lists for ML tensors, became a performance question, > > and > > > > is > > > > > > now > > > > > > > about whether vectors are mathematically valid. Each step made > > sense, > > > > > > but I > > > > > > > don't think we have consensus on where we've ended up? > > > > > > > > > > > > > > I think there is much useful work in what we left behind. Many > ML > > > > > vectors > > > > > > > aren't embeddings for search. E.g. Spark's feature vectors, > > dense and > > > > > > > sparse, routinely use NaN to mark missing values, and quantised > > > > formats > > > > > > > pack codes with scales in ways Parquet can't describe today. > > > > > ML-oriented > > > > > > > formats like Lance are moving towards tensors, quantisation and > > > > > indexes. > > > > > > A > > > > > > > type that told engines how to interpret that kind of data would > > > > enable > > > > > > new > > > > > > > things. An annotation on LIST<FLOAT> doesn't. > > > > > > > > > > > > > > My big concern with the current focus on embeddings needing a > > logical > > > > > > type > > > > > > > is compatibility. Today LIST<FLOAT> is the common format for > > > > > embeddings, > > > > > > > and Spark, BigQuery, DuckDB and pretty much every other library > > I can > > > > > > think > > > > > > > of offer distance functions on plain lists, so any engine can > > use any > > > > > > file. > > > > > > > If engines start requiring a VECTOR annotation, existing files > > and > > > > > > anything > > > > > > > rewritten by a tool that doesn't know the type stop working. If > > they > > > > > > don't > > > > > > > require it, the annotation adds nothing. We've seen this before > > with > > > > > > > strings written without the UTF8 annotation, which engines > still > > have > > > > > to > > > > > > > cope with years later. > > > > > > > > > > > > > > On invariants, I agree some users may want the writer to check > > what > > > > > they > > > > > > > write as they write it, but that can be a writer option rather > > than a > > > > > > type. > > > > > > > Readers that want evidence could get it from extra statistics > or > > > > > > metadata, > > > > > > > though careful readers will still check on read. > > > > > > > > > > > > > > If a finite fixed-length vector contract is useful to table > > formats, > > > > > > > perhaps Iceberg is the place for it, with Parquet storing plain > > > > lists. > > > > > > I'd > > > > > > > raise the same worry there, though: if tables start rejecting > > > > untagged > > > > > > > columns, we've just moved the split up a layer. > > > > > > > > > > > > > > Best, > > > > > > > Will > > > > > > > > > > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <[email protected]> > wrote: > > > > > > > > > > > > > > > I agree with much of what Dan said, but I'll add some to this > > > > thread > > > > > as > > > > > > > > well. > > > > > > > > > > > > > > > > First, I want to address the arguments that were pointed out: > > > > > > > > 1. Other vector databases have finiteness requirements > > > > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the > > > > > bandwagon > > > > > > > > 3. A general solution will take too long, finiteness will > make > > > > faster > > > > > > > > progress > > > > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will > be > > in > > > > the > > > > > > > > vector > > > > > > > > > > > > > > > > I don't think these represent our actual motivation or > > position, > > > > > which > > > > > > > > makes them function as a straw-man. This is probably not what > > was > > > > > > > > intended, but I want to make sure that the misunderstanding > > > > > > > > doesn't continue. For one thing, we aren't saying that we > need > > > > finite > > > > > > > > vectors because they are "hot stuff". > > > > > > > > > > > > > > > > We want to build tools that use them on top of Parquet. I > > think it > > > > > > should > > > > > > > > also be clear that we aren't advocating for finite elements > > blindly > > > > > > just > > > > > > > > because other engines do (cargo cult design!). There are real > > > > > > > > interoperability issues if we choose to go a different > > direction so > > > > > > this > > > > > > > is > > > > > > > > a "Chesterton's Fence" situation. It's important to > understand > > why > > > > > > > > those databases > > > > > > > > made that choice. I've also pointed this out as confirmation > > that > > > > > we're > > > > > > > > going down a reasonable and established path. > > > > > > > > > > > > > > > > The third argument sounds like a simple misunderstanding. > We're > > > > > > pursuing > > > > > > > a > > > > > > > > logical type right now because that's forward compatible and > we > > > > want > > > > > to > > > > > > > > label the data properly and set expectations for the type. > > Physical > > > > > > type > > > > > > > > changes break forward compatibility and need to be released > in > > a > > > > new > > > > > > > format > > > > > > > > version . I think this is the source of the confusion since > it > > has > > > > > been > > > > > > > in > > > > > > > > the same discussion, but the choice about finiteness is > > separate. > > > > > > > > > > > > > > > > That brings us to the last argument, whether there is value > in > > > > > writers > > > > > > > > participating in guarantees. I think there's value in adding > > > > > > requirements > > > > > > > > for writers. It's up to readers how they want to use those > > > > > > requirements, > > > > > > > > but a writer requirement allows a policy of "garbage in > garbage > > > > out" > > > > > > > rather > > > > > > > > than being forced to handle bad data. Why would a reader need > > to > > > > run > > > > > > > > validation on every element just in case a writer had a bug? > > And > > > > why > > > > > > > should > > > > > > > > the potential for writer bugs mean that it has to be the > > reader's > > > > > > > problem? > > > > > > > > If a writer produces bad data, we can absolutely reject it > and > > tell > > > > > the > > > > > > > > writer to fix the bug. > > > > > > > > > > > > > > > > I'm also puzzled by the assertion that writer guarantees have > > > > value, > > > > > > but > > > > > > > > only if there are other, more specific guarantees in > addition. > > This > > > > > > seems > > > > > > > > to cede the point that a contract between readers and writers > > makes > > > > > > > sense, > > > > > > > > but only if it is a _specific_ contract. But the proposed > > additions > > > > > > don't > > > > > > > > make sense in the same way. > > > > > > > > > > > > > > > > Take, for example, the suggestion to also disallow > zero-length > > > > > vectors. > > > > > > > At > > > > > > > > a glance this seems similar, but I think it assumes a > specific > > > > > context > > > > > > > and > > > > > > > > is better suited to be a constraint for some use cases rather > > than > > > > > part > > > > > > > of > > > > > > > > what is considered valid for the type. That's because > > zero-length > > > > > > vectors > > > > > > > > are valid vectors with a length. You can calculate Euclidean > or > > > > > > Manhattan > > > > > > > > distance and use them in many contexts. They don't work if > > you're > > > > > > > > normalizing to a length of 1, but it's a choice to normalize > > and > > > > use > > > > > a > > > > > > > dot > > > > > > > > product to calculate cosine distance. And at this point, > we're > > > > > talking > > > > > > > > about the usefulness of restrictions for the type, not a > > Platonic > > > > > > ideal. > > > > > > > > > > > > > > > > I think the restriction that makes the most sense is that > > vectors > > > > > have > > > > > > a > > > > > > > > length and can be compared in distance calculations. 0-length > > > > vectors > > > > > > are > > > > > > > > valid because they point somewhere and have a length. We > > shouldn't > > > > > > > restrict > > > > > > > > uses to only cosine distance over normalized vectors. Those > > are use > > > > > > case > > > > > > > > dependent choices and there are other valid ones. I also > don't > > > > think > > > > > > that > > > > > > > > the model used to produce a vector is part of the vector > type, > > just > > > > > > like > > > > > > > > the method used to produce a metric doesn't define its type: > > cosine > > > > > > > > similarity produces values between -1 and 1, but we store it > > in a > > > > > > > `float`, > > > > > > > > not a `float(cosine)`. > > > > > > > > > > > > > > > > Why does it make sense to have a finiteness restriction as > > part of > > > > > the > > > > > > > > type? Because that's what a (mathematical) vector is. A > simple > > > > search > > > > > > > gives > > > > > > > > a pretty common definition: "A vector is an object that has > > both > > > > > > > magnitude > > > > > > > > and a direction". If a dimension/element is NaN, null, or > > infinite, > > > > > the > > > > > > > > vector doesn't point anywhere and it doesn't have a > magnitude. > > Not > > > > > only > > > > > > > can > > > > > > > > it not be used, it doesn't meet the most basic definition of > > the > > > > > type. > > > > > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Daniel, > > > > > > > > > > > > > > > > > > alright, so no physical optimizations intended. We only > want > > > > this > > > > > > for > > > > > > > > > semantic reasons, not for any significant physical > > optimization. > > > > If > > > > > > > we're > > > > > > > > > all on the same page here, that is good 👍. > > > > > > > > > > > > > > > > > > To phrase my remaining problem with this more concretely: > > > > > > > > > *Let's say we add this type. What cool new stuff can we now > > do > > > > that > > > > > > we > > > > > > > > > couldn't do before?* > > > > > > > > > That is the litmus test that a type has to pass to be > useful > > > > IMHO. > > > > > > And > > > > > > > > here > > > > > > > > > I can't come up with a tangible use case that wasn't > possible > > > > > before. > > > > > > > > > Especially if we one day have fixed size lists in Parquet, > > the > > > > > > > remaining > > > > > > > > > niche for this type becomes so small, that the added > > complexity > > > > > isn't > > > > > > > > worth > > > > > > > > > the benefit IMHO. > > > > > > > > > > > > > > > > > > As I guess most people on here are active maintainers of a > > > > Parquet > > > > > > > > > implementation: Concretely, what would you be using this > > type for > > > > > if > > > > > > we > > > > > > > > had > > > > > > > > > it? Would you allow math only on fields tagged with this > > type? > > > > What > > > > > > if > > > > > > > > that > > > > > > > > > math yields INF or NaN? Not allowed? > > > > > > > > > > > > > > > > > > Especially if we also discuss having both VECTOR and > > > > FINITE_VECTOR. > > > > > > How > > > > > > > > > will you guys wire this up with the type systems of your > > engines? > > > > > Do > > > > > > > your > > > > > > > > > engines also have two types here? Do you type-decay a > > > > FINITE_VECTOR > > > > > > to > > > > > > > a > > > > > > > > > VECTOR once someone does a multiplication that could yield > > INF? > > > > By > > > > > > > > itself, > > > > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a > > type > > > > > > system > > > > > > > > > perspective, it comes with a lot of complexity. Do the > > benefits > > > > > > really > > > > > > > > > outweigh the complexity here? That's why I would rather > have > > the > > > > > bare > > > > > > > > > minimum: A fixed size array (list) of floats or at least > > just a > > > > > > vector > > > > > > > > > without finite. Then all these detailed questions go away. > > > > > > > > > > > > > > > > > > I'll take this one step further... Why is there any debate > > about > > > > > this > > > > > > > at > > > > > > > > > > all? Why can't a type simply be added to the > > *specification* as > > > > > an > > > > > > > > > > administrative detail and be done? > > > > > > > > > > > > > > > > > > > > > > > > > > > Andrew, > > > > > > > > > > > > > > > > > > Parquet is supposed to be a lingua franca of data. The > > > > > specification > > > > > > is > > > > > > > > the > > > > > > > > > authoritative truth for this language and all > > implementations are > > > > > > > > expected > > > > > > > > > to implement as much as possible from this spec, so > anything > > in > > > > the > > > > > > > spec > > > > > > > > > does causes implementation effort. So just adding anything > to > > > > this > > > > > > spec > > > > > > > > > without debate is not a good idea. It is even the other way > > > > around: > > > > > > We > > > > > > > > need > > > > > > > > > to have way higher rigor with the spec than we need to have > > with > > > > > any > > > > > > > > > implementations. Implementations can be changed, the spec > > needs > > > > to > > > > > be > > > > > > > > > stable. > > > > > > > > > > > > > > > > > > A tag-only type that doesn't come with any restrictions can > > be > > > > > added > > > > > > > with > > > > > > > > > little effort. But here we want to tie restrictions to it > (no > > > > > > NaN/Inf), > > > > > > > > so > > > > > > > > > we would expect implementations to enforce these. So there > is > > > > some > > > > > > > > > implementation effort expected here. > > > > > > > > > > > > > > > > > > Also, even if we only wanted a tag type, I would still > argue > > that > > > > > you > > > > > > > > > shouldn't add those without a debate. What if A wants XML, > B > > > > wants > > > > > > > YAML, > > > > > > > > C > > > > > > > > > wants quaternions, D wants address, E wants phone number. > If > > we > > > > > just > > > > > > > > added > > > > > > > > > any types that are "just tags", Parquet logical types would > > just > > > > > > > become a > > > > > > > > > trash dump of any type that anyone ever thought to be > > useful. So > > > > > > > rigorous > > > > > > > > > assessment of each new type candidate is necessary. > > > > > > > > > > > > > > > > > > Finally, many engines couple parquet with another type > system > > > > > (e.g., > > > > > > > > Arrow > > > > > > > > > or SQL). So even with a tag only type, you at least still > > have > > > > the > > > > > > > > > implementation burden of wiring this up with your engine's > > type > > > > > > system. > > > > > > > > > > > > > > > > > > So, just because logical types are "just" logical, that > > doesn't > > > > > mean > > > > > > we > > > > > > > > can > > > > > > > > > add them without any rigor. > > > > > > > > > > > > > > > > > > > > > > > > > > > A few naive questions: > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow > > > > efficient > > > > > > > > > > encodings like ALP? > > > > > > > > > > 2. If the physical representation allows values that make > > > > > encodings > > > > > > > > like > > > > > > > > > > ALP impossible, can a tighter Logical type allow > different > > > > > > encodings? > > > > > > > > > > 3. Can we RLE or binpack a vector? > > > > > > > > > > > > > > > > > > > > > > > > > > > The simple general answer here is that all these things do > > not > > > > > > require > > > > > > > a > > > > > > > > > logical type. Writers can already do any such optimization > > if the > > > > > > > actual > > > > > > > > > data contains no NaNs & Infs. All these physical > > optimizations > > > > just > > > > > > > > require > > > > > > > > > no actual NaNs/Infs to be present in a page, they don't > need > > a > > > > > > > guarantee > > > > > > > > > that the whole file won't contain any. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Jan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb < > > > > > [email protected] > > > > > > >: > > > > > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > > > > > Glad to see a Vector type being added. I have a few use > > cases I > > > > > > want > > > > > > > to > > > > > > > > > > pursue once we have an agreed upon definition. > > > > > > > > > > > > > > > > > > > > In regards to that definition discussion. I feel > > comfortable > > > > with > > > > > > the > > > > > > > > > idea > > > > > > > > > > of the semantic meaning Dan raises though I can’t shake a > > > > slight > > > > > > > > > discomfort > > > > > > > > > > (maybe FOMO) with limiting values in the physical > > definition. > > > > > > > > > > > > > > > > > > > > A few naive questions: > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow > > > > efficient > > > > > > > > > > encodings like ALP? > > > > > > > > > > 2. If the physical representation allows values that make > > > > > encodings > > > > > > > > like > > > > > > > > > > ALP impossible, can a tighter Logical type allow > different > > > > > > encodings? > > > > > > > > > > 3. Can we RLE or binpack a vector? > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > Kurtis C. Wright > > > > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hey Jan, > > > > > > > > > > > > > > > > > > > > > > There's a lot that I disagree with respect to the > > framing of > > > > > > these > > > > > > > > > > > arguments and how we should be thinking about > introducing > > > > > vector > > > > > > > as a > > > > > > > > > > > logical type. There's a repeated tendency to let the > > > > physical > > > > > > > > > > > representation or performance drive discussion around > > types > > > > as > > > > > > > > opposed > > > > > > > > > to > > > > > > > > > > > what it represents. > > > > > > > > > > > > > > > > > > > > > > In parquet, establishing semantic meaning is the > purpose > > of > > > > > > logical > > > > > > > > > > types. > > > > > > > > > > > The logical type separates how to interpret a value > from > > its > > > > > > > physical > > > > > > > > > > > representation. We can represent a timestamp with > INT64 > > or > > > > > even > > > > > > a > > > > > > > > > string > > > > > > > > > > > value, so why have a type for it? Even though a > > > > > FIXED_SIZED_LIST > > > > > > > and > > > > > > > > > > VECTOR > > > > > > > > > > > may have the exact same representation and performance, > > they > > > > > have > > > > > > > > > > different > > > > > > > > > > > semantic meaning. Databases and engines across the > > ecosystem > > > > > > > already > > > > > > > > > > > recognize this differentiation and have separate > > > > > representations. > > > > > > > > The > > > > > > > > > > > choice to separate these isn't because the discussion > is > > too > > > > > > > tedious; > > > > > > > > > > > they're fundamentally different things. > > > > > > > > > > > > > > > > > > > > > > The finite requirement is what makes a vector a > > reflection of > > > > > > what > > > > > > > it > > > > > > > > > > > represents. As we've discussed in the thread, > non-finite > > > > > values > > > > > > in > > > > > > > > the > > > > > > > > > > > representation violate the properties of a vector and > the > > > > value > > > > > > > > ceases > > > > > > > > > to > > > > > > > > > > > be a vector. On the other hand, a 0 vector is still a > > vector > > > > > > (you > > > > > > > > can > > > > > > > > > > > argue the utility of it, but it is a valid vector). > > > > > > > Implementations > > > > > > > > > can > > > > > > > > > > > choose what they validate or how to handle execution > > errors; > > > > > > that's > > > > > > > > an > > > > > > > > > > > implementation concern, not a requirement. Operations > > > > > performed > > > > > > > on a > > > > > > > > > > > vector depend on the context in which they are used, > but > > the > > > > > > > > > > representation > > > > > > > > > > > doesn't need to solve scenarios where that application > > > > doesn't > > > > > > make > > > > > > > > > > sense. > > > > > > > > > > > > > > > > > > > > > > The guarantees or context needed to use a vector do not > > need > > > > to > > > > > > be > > > > > > > > > > inherent > > > > > > > > > > > in the type for it to be a useful type. We don't need > > the > > > > > > currency > > > > > > > > > used > > > > > > > > > > > for decimal or whether a floating point is a normalized > > value > > > > > as > > > > > > > part > > > > > > > > > of > > > > > > > > > > > the type to make use of those types. This conflates > the > > use > > > > > > cases > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > type system and unnecessarily attempts to draw the > > context > > > > into > > > > > > the > > > > > > > > > type > > > > > > > > > > > definition. Similarly, you don't need to capture > > > > > normalization, > > > > > > > > model, > > > > > > > > > > > etc. as part of the type for it to be useful. > > > > > > > > > > > > > > > > > > > > > > So the reason we want to add a vector type (or > > FINITE_VECTOR) > > > > > is > > > > > > > > > because > > > > > > > > > > we > > > > > > > > > > > want to store vectors with the semantic meaning and > > > > > requirements > > > > > > > > based > > > > > > > > > on > > > > > > > > > > > what they represent. > > > > > > > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >> A bit late to the party, but here we go: > > > > > > > > > > >> > > > > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR > type > > as > > > > it > > > > > is > > > > > > > > > > currently > > > > > > > > > > >> being discussed will not be very useful and just > > increase > > > > > > > > > implementation > > > > > > > > > > >> complexity. The arguments for the type are all quite > > weak > > > > > > (details > > > > > > > > see > > > > > > > > > > >> below). The FINITE part doesn't seem to yield any > value > > and > > > > is > > > > > > > not a > > > > > > > > > > useful > > > > > > > > > > >> restriction by itself. The vector (i.e. fixed length) > > part > > > > > > should > > > > > > > > > > rather be > > > > > > > > > > >> generalized instead of making a one-off thing. In > > general, > > > > the > > > > > > > > > existing > > > > > > > > > > >> types are already working well for uncompressed > > vectors. The > > > > > > real > > > > > > > > > > challenge > > > > > > > > > > >> that would bring real benefits is supporting > > quantization. > > > > > This > > > > > > is > > > > > > > > > where > > > > > > > > > > >> you get orders of magnitude improvements to storage > > space > > > > and > > > > > > > > > processing > > > > > > > > > > >> speed. > > > > > > > > > > >> > > > > > > > > > > >> Let's first discuss the FINITE part. So far, the only > > > > > arguments > > > > > > I > > > > > > > > have > > > > > > > > > > read > > > > > > > > > > >> for this are: > > > > > > > > > > >> > > > > > > > > > > >> 1. many other vector databases have it > > > > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet > > better > > > > > boast > > > > > > > > having > > > > > > > > > > >> "native" type support for vectors > > > > > > > > > > >> 3. by focussing on this common special case, we can > make > > > > rapid > > > > > > > > > progress, > > > > > > > > > > >> while a general solution would take too long to > specify > > > > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL > > will > > > > be > > > > > > in > > > > > > > > the > > > > > > > > > > >> vector > > > > > > > > > > >> > > > > > > > > > > >> the first two arguments are quite weak by themselves. > > Vector > > > > > > > > databases > > > > > > > > > > are > > > > > > > > > > >> very specialized engines. Such engines can of course > be > > very > > > > > > > > > > opinionated on > > > > > > > > > > >> their data types. A general format like Parquet must > > instead > > > > > > weigh > > > > > > > > > > whether > > > > > > > > > > >> a special type is worth the added complexity over a > more > > > > > > general, > > > > > > > > > widely > > > > > > > > > > >> applicable type. In this case, it is IMHO not at all. > > And > > > > just > > > > > > > > having > > > > > > > > > a > > > > > > > > > > >> type called "vector" for advertisement reasons to make > > > > Parquet > > > > > > > > appear > > > > > > > > > > more > > > > > > > > > > >> modern seems a non-goal in the first place. > > > > > > > > > > >> > > > > > > > > > > >> The third "time to market" argument is valid for > systems > > > > that > > > > > > > don't > > > > > > > > > > need a > > > > > > > > > > >> stable interface. It is a very dangerous attitude for > > > > formats > > > > > > like > > > > > > > > > > Parquet > > > > > > > > > > >> that must be stable long term and should have a > > > > consolidated, > > > > > > > > concise, > > > > > > > > > > >> non-redundant type system. For such long term storage > > > > formats, > > > > > > we > > > > > > > > need > > > > > > > > > > to > > > > > > > > > > >> take the time to build the right thing, as any > > shortcuts we > > > > > take > > > > > > > > will > > > > > > > > > > stay > > > > > > > > > > >> as blemishes in the format forever. If I read the > > discussion > > > > > > > > > correctly, > > > > > > > > > > >> this is actually the main argument as of now: We have > > found > > > > > that > > > > > > > it > > > > > > > > is > > > > > > > > > > hard > > > > > > > > > > >> to align on how a fixed size list type should look > > like, so > > > > we > > > > > > > want > > > > > > > > to > > > > > > > > > > go > > > > > > > > > > >> the easier route of a specialized type. This seems > like > > a > > > > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask > > > > > yourself: > > > > > > > > Would > > > > > > > > > > we > > > > > > > > > > >> want this, if Parquet already had a fixed size list > > type? > > > > > > > > > > >> > > > > > > > > > > >> So finally, the only technical argument is the fourth > > one: > > > > "It > > > > > > > > signals > > > > > > > > > > >> semantics and ensures that no bad data lands in the > > > > format". I > > > > > > see > > > > > > > > two > > > > > > > > > > big > > > > > > > > > > >> problems with this: > > > > > > > > > > >> > > > > > > > > > > >> 1. Readers *still* need to test for NaN / INF because > > > > Parquet > > > > > > > files > > > > > > > > > may > > > > > > > > > > be > > > > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer > > from > > > > > > > > physically > > > > > > > > > > >> writing NaN or INF. *So you save no single line of > code > > in a > > > > > > > reader. > > > > > > > > > You > > > > > > > > > > >> save no branches, as you still need to handle the > > INF/NaN > > > > > case.* > > > > > > > So > > > > > > > > > this > > > > > > > > > > >> doesn't help with performance at all. The only > > difference > > > > is, > > > > > > that > > > > > > > > the > > > > > > > > > > >> error message changes from "found an invalid INF value > > in > > > > the > > > > > > > data" > > > > > > > > to > > > > > > > > > > >> "Your parquet file seems to be corrupt". It is > actually > > > > quite > > > > > > the > > > > > > > > > > opposite, > > > > > > > > > > >> readers and writers get more complex having to handle > > this > > > > new > > > > > > > type. > > > > > > > > > > >> > > > > > > > > > > >> If you now argue that it's not about performance, it's > > about > > > > > > > defined > > > > > > > > > > >> semantics and erroring out early in writers, then the > > second > > > > > > > problem > > > > > > > > > > >> arises: > > > > > > > > > > >> > > > > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one > of > > > > > > multiple > > > > > > > > > > >> guarantees that you need for sensible vector > > processing. If > > > > > "no > > > > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I > would > > > > agree > > > > > > > that > > > > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the > only > > > > > > guarantee > > > > > > > > you > > > > > > > > > > need, > > > > > > > > > > >> instead, depending on the distance metric, you also > > need the > > > > > > > > following > > > > > > > > > > >> guarantees. E.g.: > > > > > > > > > > >> > > > > > > > > > > >> - No vector may be the 0 vector, otherwise, e.g., > > cosine > > > > > > > > similarity > > > > > > > > > is > > > > > > > > > > >> undefined. Cosine similarity is often used as > > default, so > > > > > this > > > > > > > is > > > > > > > > > > >> actually > > > > > > > > > > >> quite important in practice. > > > > > > > > > > >> - If we want to use a dot product, then all vectors > > need > > > > to > > > > > > > > > guarantee > > > > > > > > > > a > > > > > > > > > > >> length of 1. This would actually be quite useful for > > > > > > performance > > > > > > > > > > >> reasons. > > > > > > > > > > >> - The vectors must come from the same model as the > > vector > > > > > used > > > > > > > for > > > > > > > > > the > > > > > > > > > > >> search. Otherwise, the search yields a numeric > > result, but > > > > > > that > > > > > > > > > result > > > > > > > > > > >> has > > > > > > > > > > >> no sensible meaning. > > > > > > > > > > >> > > > > > > > > > > >> So, what use is a type that just guarantees part of > the > > > > things > > > > > > you > > > > > > > > > need > > > > > > > > > > for > > > > > > > > > > >> vector search but not others? You still need > additional > > > > > > guarantees > > > > > > > > in > > > > > > > > > > >> addition to that type. What makes INF/NaN so important > > in > > > > > > > comparison > > > > > > > > > to, > > > > > > > > > > >> say, "no 0 vector present", that it warrants an own > > type, > > > > > while > > > > > > > the > > > > > > > > > > other > > > > > > > > > > >> guarantees do not? If we really want this type as a > > basis > > > > for > > > > > > > > > > cross-engine > > > > > > > > > > >> sharing of vectors, then it needs to give *all* > > guarantees. > > > > > > > > > > >> > > > > > > > > > > >> I'm not against a specialized vector type if it really > > > > carries > > > > > > > *all* > > > > > > > > > > >> information to be useful by itself. If it carried all > > that > > > > > > > > > information, > > > > > > > > > > >> then a reading engine could indeed validate that only > > > > > meaningful > > > > > > > > > vector > > > > > > > > > > >> searches are executed. But as long as it is just a > > > > half-baked > > > > > > > thing, > > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > >> not useful by itself and just a burden on each > > implementer > > > > and > > > > > > > > > > maintainer. > > > > > > > > > > >> If we need extra information anyway, why not just > stick > > to a > > > > > > > general > > > > > > > > > > type > > > > > > > > > > >> like VECTOR (without FINITE). > > > > > > > > > > >> > > > > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself > (or > > > > tell > > > > > > us) > > > > > > > > what > > > > > > > > > > >> advantage you will *really* get over just a > (potentially > > > > fixed > > > > > > > size) > > > > > > > > > > List > > > > > > > > > > >> of Float? As discussed, it's not performance. It's > also > > not > > > > a > > > > > > > > semantic > > > > > > > > > > >> marker that is sufficient by itself. It's just a small > > hint > > > > > that > > > > > > > > > allows > > > > > > > > > > >> writers to error out once they see a NaN or INF in the > > > > input, > > > > > > > while > > > > > > > > > they > > > > > > > > > > >> will still happily write a bogus 0 vector. This seems > > to be > > > > of > > > > > > > zero > > > > > > > > > > >> benefit. > > > > > > > > > > >> > > > > > > > > > > >> Anecdotally, we currently handle petabytes of vectors > in > > > > > > > production > > > > > > > > in > > > > > > > > > > >> Data360. We simply store them in a List of Float, with > > an > > > > > > > optimized > > > > > > > > > > R-Level > > > > > > > > > > >> reader implementation that rapidly checks that all > > vectors > > > > are > > > > > > the > > > > > > > > > same > > > > > > > > > > >> length. This solution works just fine. The overhead of > > > > R-level > > > > > > > > > decoding > > > > > > > > > > is > > > > > > > > > > >> barely measurable. The check is super fast and > R-levels > > are > > > > > tiny > > > > > > > in > > > > > > > > > > >> comparison to the payload vectors, so also the added > > I/O is > > > > > > > > > negligible. > > > > > > > > > > >> This is also why I'm so firm in my sentiment: We are > > > > running a > > > > > > > > > > production > > > > > > > > > > >> grade system processing trillions of vectors right now > > and > > > > see > > > > > > no > > > > > > > > big > > > > > > > > > > >> issues with the types that Parquet currently provides. > > > > > > > > > > >> > > > > > > > > > > >> As I don't just want to be a naysayer and point out > > problems > > > > > > with > > > > > > > > the > > > > > > > > > > >> current design, here are the directions that I would > > propose > > > > > > > indeed: > > > > > > > > > > >> > > > > > > > > > > >> - a general fixed size list type. Don't do shortcuts > > just > > > > > > > because > > > > > > > > > the > > > > > > > > > > >> discussion is too tedious. > > > > > > > > > > >> - a vector type that indeed carries enough > > information to > > > > be > > > > > > > > really > > > > > > > > > > >> useful for vector search by itself. (e.g., model > name > > / > > > > > > version, > > > > > > > > > > >> optional > > > > > > > > > > >> guarantees for no 0 vector / normalized vector > > length). > > > > With > > > > > > > such > > > > > > > > a > > > > > > > > > > >> type, > > > > > > > > > > >> you could really do well defined vector searches > even > > if > > > > the > > > > > > > > writing > > > > > > > > > > >> engine > > > > > > > > > > >> is completely different to the reading one. > > > > > > > > > > >> - Standardize quantizations. This will be a real > game > > > > > changer > > > > > > > for > > > > > > > > > > >> performance. > > > > > > > > > > >> > > > > > > > > > > >> Cheers, > > > > > > > > > > >> Jan > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu < > > > > > > > > [email protected] > > > > > > > > > >: > > > > > > > > > > >>> > > > > > > > > > > >>> I think separating them into FINITE_VECTOR and > > > > > > FIXED_LENGTH_LIST > > > > > > > > is a > > > > > > > > > > >> good > > > > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR > > looks > > > > > > more > > > > > > > > > > natural. > > > > > > > > > > >>> Should we move forward with proposing > > FINITE_VECTOR/VECTOR > > > > > > first? > > > > > > > > > > >>> > > > > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via > > dev < > > > > > > > > > > >>> [email protected]> wrote: > > > > > > > > > > >>> > > > > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to > > keep > > > > > the > > > > > > > name > > > > > > > > > the > > > > > > > > > > >>> same > > > > > > > > > > >>>> as other engines/type systems. If we want a > > non-finite one > > > > > in > > > > > > > the > > > > > > > > > > >> future > > > > > > > > > > >>> we > > > > > > > > > > >>>> can name that something else or we add fixed size > list > > > > that > > > > > > will > > > > > > > > > serve > > > > > > > > > > >>> the > > > > > > > > > > >>>> same purpose without a special name. > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > >>>> > > > > > > > > > > >>>>> A logical type would: > > > > > > > > > > >>>>> - Describe data for users and readers to understand > > what > > > > > the > > > > > > > file > > > > > > > > > > >>>> contains > > > > > > > > > > >>>>> - Signal that repetition and definition levels will > > > > > reflect a > > > > > > > > fixed > > > > > > > > > > >>>> number > > > > > > > > > > >>>>> of elements, which implementations can use to > > optimize > > > > > > decoding > > > > > > > > > > >>>>> - Guarantee that there are no null elements > (because > > the > > > > > type > > > > > > > > > cannot > > > > > > > > > > >> be > > > > > > > > > > >>>>> applied to a list with optional elements) > > > > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are > NaN > > or > > > > > > > Infinity > > > > > > > > > > >> values > > > > > > > > > > >>>> so > > > > > > > > > > >>>>> that the vectors have a finite length and are > > suitable > > > > for > > > > > > > > distance > > > > > > > > > > >>>>> calculations > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> The finite value restriction is a restriction on > > writers > > > > to > > > > > > > > prevent > > > > > > > > > > >> bad > > > > > > > > > > >>>>> data from being written. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Ignoring logical types that are not understood is a > > rule > > > > > for > > > > > > > > > > >> readers. A > > > > > > > > > > >>>>> reader can handle vectors and pass it on to another > > > > layer. > > > > > > For > > > > > > > > > > >>> instance, > > > > > > > > > > >>>> a > > > > > > > > > > >>>>> Parquet reader that doesn't support geometry passes > > > > values > > > > > as > > > > > > > > > > >> `binary`. > > > > > > > > > > >>>>> That binary value can be passed to a geometry UDF, > > even > > > > if > > > > > > the > > > > > > > > > > >>> underlying > > > > > > > > > > >>>>> engine and Parquet reader doesn't have support. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can > > > > similarly > > > > > > > pass a > > > > > > > > > > >> list > > > > > > > > > > >>> of > > > > > > > > > > >>>>> floats along to a UDF. But it is still important to > > the > > > > > user > > > > > > > that > > > > > > > > > bad > > > > > > > > > > >>>>> values are not present in the data. In your > example, > > bad > > > > > > values > > > > > > > > are > > > > > > > > > > >>>> vectors > > > > > > > > > > >>>>> that cause the centroid calculation to fail, like > > vectors > > > > > > with > > > > > > > > > > >> infinite > > > > > > > > > > >>>>> length. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Ryan > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via > dev > > < > > > > > > > > > > >>>>> [email protected]> > > > > > > > > > > >>>>> wrote: > > > > > > > > > > >>>>> > > > > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR > > annotation > > > > on > > > > > > > > > > >>> LIST<FLOAT>? > > > > > > > > > > >>>>>> Specifically, what does it make possible that's > > > > impossible > > > > > > > > today, > > > > > > > > > > >> and > > > > > > > > > > >>>>> what > > > > > > > > > > >>>>>> does it make easier that's currently hard? > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like > > every > > > > > > engine > > > > > > > to > > > > > > > > > > >>> offer > > > > > > > > > > >>>>>> vector functions on lists of floats without > needing > > a > > > > new > > > > > > > > logical > > > > > > > > > > >>> type. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector > > type > > > > > that > > > > > > > > > > >>> describes > > > > > > > > > > >>>>> how > > > > > > > > > > >>>>>> to unpack and scale composite representations > > (quantised > > > > > > codes > > > > > > > > > plus > > > > > > > > > > >>>>> scales, > > > > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going. > That > > > > adds > > > > > > > > > > >> information > > > > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> > doesn't: > > > > > engines > > > > > > > > > already > > > > > > > > > > >>>>>> understand the physical type, and many already > offer > > > > > vector > > > > > > > > > > >> functions > > > > > > > > > > >>>> on > > > > > > > > > > >>>>>> it. I don't see a performance benefit, and > > NaN/null/Inf > > > > > > checks > > > > > > > > are > > > > > > > > > > >>>> cheap > > > > > > > > > > >>>>>> enough that engines should do them anyway. The > label > > > > also > > > > > > > > doesn't > > > > > > > > > > >> say > > > > > > > > > > >>>>> what > > > > > > > > > > >>>>>> you can do with the column. Knowing that a column > > holds > > > > > > > > embeddings > > > > > > > > > > >>>>> doesn't > > > > > > > > > > >>>>>> say how they can be used, just as zipcode and age > > are > > > > both > > > > > > > > > integers > > > > > > > > > > >>> but > > > > > > > > > > >>>>>> multiplying them is meaningless. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> My concern is the opposite effect. We're > > establishing > > > > that > > > > > > > > engines > > > > > > > > > > >>>> should > > > > > > > > > > >>>>>> ignore logical types they don't understand. So a > > tool > > > > that > > > > > > > > doesn't > > > > > > > > > > >>> know > > > > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let > the > > user > > > > > > > compute > > > > > > > > > > >>>>> centroids > > > > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If > > other > > > > > > engines > > > > > > > > > > >>> restrict > > > > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns, > > that > > > > file > > > > > > > stops > > > > > > > > > > >>>> working > > > > > > > > > > >>>>>> with them, through no fault of the user. An > > annotation > > > > > that > > > > > > > > > doesn't > > > > > > > > > > >>>>> change > > > > > > > > > > >>>>>> what the data is shouldn't be able to make it > > unusable. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Am I missing a use case? > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Best, Will > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu < > > > > > > [email protected]> > > > > > > > > > > >> wrote: > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of > > > > values > > > > > to > > > > > > > > many > > > > > > > > > > >>> use > > > > > > > > > > >>>>>> cases. > > > > > > > > > > >>>>>>> It is would be great to make progress on the > > > > > FINITE_VECTOR, > > > > > > > and > > > > > > > > > > >> not > > > > > > > > > > >>>> to > > > > > > > > > > >>>>>> fit > > > > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST. > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue < > > > > > [email protected]> > > > > > > > > > > >> wrote: > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>>> Going through this thread again, it looks like > > there > > > > > are a > > > > > > > few > > > > > > > > > > >>>> things > > > > > > > > > > >>>>>>> that > > > > > > > > > > >>>>>>>> people generally agree on: > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> - In the short term, we want to introduce a > > logical > > > > type > > > > > > or > > > > > > > > > > >>> types; > > > > > > > > > > >>>> a > > > > > > > > > > >>>>>>> better > > > > > > > > > > >>>>>>>> physical representation can be done later. We > can > > > > focus > > > > > on > > > > > > > > > > >>> logical > > > > > > > > > > >>>>>> types. > > > > > > > > > > >>>>>>>> - There is precedent in logical types for > > restricting > > > > > > > values, > > > > > > > > > > >>> like > > > > > > > > > > >>>>> UTF8 > > > > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I > think > > we > > > > > also > > > > > > > > agree > > > > > > > > > > >>>> that > > > > > > > > > > >>>>> we > > > > > > > > > > >>>>>>>> could restrict vector elements to non-nested or > > > > > primitive > > > > > > > > > > >> types. > > > > > > > > > > >>>>>>>> - There are many examples of databases that > > restrict > > > > > their > > > > > > > > > > >> vector > > > > > > > > > > >>>>> types > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>> non-null, finite elements. > > > > > > > > > > >>>>>>>> - Some people want to have write-time guarantees > > that > > > > > > > vectors > > > > > > > > > > >>> have > > > > > > > > > > >>>>> only > > > > > > > > > > >>>>>>>> non-null, finite elements > > > > > > > > > > >>>>>>>> - Some people want to extend this work to > include > > > > > tensors, > > > > > > > > > > >>> shorter > > > > > > > > > > >>>>>>> floating > > > > > > > > > > >>>>>>>> point representations, variable lengths, NaN or > > > > Infinity > > > > > > > > > > >>> elements, > > > > > > > > > > >>>>> etc. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two > > > > halves > > > > > of > > > > > > > the > > > > > > > > > > >>>>>> community > > > > > > > > > > >>>>>>>> that care about different things. That just > means > > we > > > > > > should > > > > > > > > > > >> have > > > > > > > > > > >>>>>> multiple > > > > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type > > that > > > > > allows > > > > > > > NaN > > > > > > > > > > >>>>> values, > > > > > > > > > > >>>>>>> and > > > > > > > > > > >>>>>>>> it's also reasonable for those of us building > > > > databases, > > > > > > > table > > > > > > > > > > >>>>>> standards, > > > > > > > > > > >>>>>>>> and vector indexes to care about guarantees that > > > > > elements > > > > > > > are > > > > > > > > > > >>>> finite. > > > > > > > > > > >>>>>>>> There's more than enough examples to show that > > this > > > > is a > > > > > > > > > > >>> legitimate > > > > > > > > > > >>>>> use > > > > > > > > > > >>>>>>>> case. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Since I don't see the concerns of the different > > use > > > > > cases > > > > > > > > > > >>> changing, > > > > > > > > > > >>>>> it > > > > > > > > > > >>>>>>>> looks like we need to find a compromise to move > > > > > forward. I > > > > > > > > > > >>> propose > > > > > > > > > > >>>>> that > > > > > > > > > > >>>>>>> we > > > > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a > > > > > > > > > > >> FIXED_LENGTH_LIST > > > > > > > > > > >>>> or > > > > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN > > > > values. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong > > reason > > > > > to > > > > > > > try > > > > > > > > > > >> to > > > > > > > > > > >>>> fit > > > > > > > > > > >>>>>>> these > > > > > > > > > > >>>>>>>> use cases into a single type? What if that means > > not > > > > > > making > > > > > > > > > > >>>> progress? > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Ryan > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp > Fischbeck > > via > > > > > dev > > > > > > < > > > > > > > > > > >>>>>>>> [email protected]> wrote: > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with > > pulling > > > > > out > > > > > > > the > > > > > > > > > > >>>>>> “NaN/Inf” > > > > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my > > > > > preference), > > > > > > or > > > > > > > > > > >>>> putting > > > > > > > > > > >>>>>> no > > > > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all > > IEEE > > > > > 754 > > > > > > > > > > >> float > > > > > > > > > > >>>>>> numbers. > > > > > > > > > > >>>>>>>> If > > > > > > > > > > >>>>>>>>> some top-level application wants to put > > additional > > > > > > > > > > >> constraints > > > > > > > > > > >>> on > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>>> data > > > > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, > it > > can > > > > > > still > > > > > > > do > > > > > > > > > > >>> so > > > > > > > > > > >>>>> one > > > > > > > > > > >>>>>>>> layer > > > > > > > > > > >>>>>>>>> above. It seems like this was the biggest point > > of > > > > > > > discussion > > > > > > > > > > >>>>>>> remaining. > > > > > > > > > > >>>>>>>>> All these options work for me, and we should > move > > > > > forward > > > > > > > > > > >> with > > > > > > > > > > >>>> this > > > > > > > > > > >>>>>>>>> proposal. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Concerning additional element types like bf16, > > they > > > > can > > > > > > be > > > > > > > > > > >>>>> introduced > > > > > > > > > > >>>>>>> as > > > > > > > > > > >>>>>>>>> Parquet types (at least in the current > LIST-based > > > > spec > > > > > > > > > > >>> proposal). > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Concerning Will’s suggestion around > representing > > > > VECTOR > > > > > > as > > > > > > > a > > > > > > > > > > >>>>> variable > > > > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata > > > > > restriction: I > > > > > > > > > > >> think > > > > > > > > > > >>>>> that > > > > > > > > > > >>>>>> a > > > > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides > > enough > > > > > > > benefits > > > > > > > > > > >> by > > > > > > > > > > >>>>>>>> describing > > > > > > > > > > >>>>>>>>> the semantics of the data, and allowing to > > capture > > > > > future > > > > > > > > > > >>> changes > > > > > > > > > > >>>>>>> around > > > > > > > > > > >>>>>>>>> statistics. In that case, the length is better > > > > captured > > > > > > as > > > > > > > an > > > > > > > > > > >>>> exact > > > > > > > > > > >>>>>>>> length > > > > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure > > about > > > > > other > > > > > > > use > > > > > > > > > > >>>> cases > > > > > > > > > > >>>>>> for > > > > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type > constraint > > > > system > > > > > > > just > > > > > > > > > > >>> for > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t > > have > > > > > > enough > > > > > > > > > > >>> value > > > > > > > > > > >>>> in > > > > > > > > > > >>>>>> my > > > > > > > > > > >>>>>>>>> view. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Best, > > > > > > > > > > >>>>>>>>> Philipp > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via > > dev < > > > > > > > > > > >>>>>>>>> [email protected]> > > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage > > beyond > > > > > > > > > > >> normalized > > > > > > > > > > >>>>>>> embeddings > > > > > > > > > > >>>>>>>>> and > > > > > > > > > > >>>>>>>>>> engines really aren't going to be able to > elide > > all > > > > > the > > > > > > > > > > >>> checks > > > > > > > > > > >>>>>>> anyway. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata > > solutions > > > > > too > > > > > > > :) > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or > > VARCHAR(n) > > > > > can't > > > > > > > > > > >>>> express > > > > > > > > > > >>>>>> this > > > > > > > > > > >>>>>>>> in > > > > > > > > > > >>>>>>>>>> Parquet. Engines can write opaque blobs they > > > > > understand > > > > > > > in > > > > > > > > > > >>> the > > > > > > > > > > >>>>>>>> metadata > > > > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info > > > > round-trips, > > > > > > but > > > > > > > > > > >>> this > > > > > > > > > > >>>>>> isn't > > > > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata > or > > new > > > > > > Thrift > > > > > > > > > > >>>> fields > > > > > > > > > > >>>>>> (my > > > > > > > > > > >>>>>>>>> pref) > > > > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also > > being > > > > > more > > > > > > > > > > >>>> broadly > > > > > > > > > > >>>>>>> useful > > > > > > > > > > >>>>>>>>> for > > > > > > > > > > >>>>>>>>>> other use cases and data types? > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the > > > > > MIN_LENGTH > > > > > > > and > > > > > > > > > > >>>>>>> MAX_LENGTH > > > > > > > > > > >>>>>>>>> for > > > > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types. > These > > > > would > > > > > > > > > > >> count > > > > > > > > > > >>>>>> logical > > > > > > > > > > >>>>>>>>> units > > > > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than > > > > bytes. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to > > > > Parquet > > > > > as > > > > > > > > > > >>>> LIST(N) > > > > > > > > > > >>>>>> with > > > > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - > > even > > > > > > > > > > >> different > > > > > > > > > > >>>>>>> engines - > > > > > > > > > > >>>>>>>>> can > > > > > > > > > > >>>>>>>>>> pick up these constraints if they support > doing > > so. > > > > > We > > > > > > > > > > >> don't > > > > > > > > > > >>>>>>> actually > > > > > > > > > > >>>>>>>>> need > > > > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and > > > > VARCHAR(N) > > > > > > to > > > > > > > > > > >> be > > > > > > > > > > >>>>>>> expressed > > > > > > > > > > >>>>>>>> on > > > > > > > > > > >>>>>>>>>> UTF8 columns etc. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> This also seems generally useful in the future > > as we > > > > > > > > > > >> approach > > > > > > > > > > >>>>>>>> expressing > > > > > > > > > > >>>>>>>>>> other tensor data shapes and types. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage > > type > > > > > > doesn't > > > > > > > > > > >>>> enforce > > > > > > > > > > >>>>>>> these > > > > > > > > > > >>>>>>>>>> constraints and engines must check them. It's > > > > > previously > > > > > > > > > > >>>>>> established > > > > > > > > > > >>>>>>>> that > > > > > > > > > > >>>>>>>>>> performance is just a software problem and > > there is > > > > no > > > > > > > > > > >>>>> fundamental > > > > > > > > > > >>>>>>>>>> performance advantage to the fixed size types > in > > > > > > > > > > >> Dremel-like > > > > > > > > > > >>>>>> formats > > > > > > > > > > >>>>>>>> like > > > > > > > > > > >>>>>>>>>> Parquet. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>> Will > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian > > Baunsgaard > > > > < > > > > > > > > > > >>>>>>>>> [email protected] > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> After listening in on the last meeting > > Wednesday, I > > > > > > have > > > > > > > > > > >>> been > > > > > > > > > > >>>>>>>> thinking > > > > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would > > share > > > > my > > > > > 2 > > > > > > > > > > >>> cents. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be > > > > finite, > > > > > > as > > > > > > > > > > >>>>> proposed > > > > > > > > > > >>>>>>> in > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>> PR > > > > > > > > > > >>>>>>>>>>> < > > https://github.com/apache/parquet-format/pull/624 > > > > >. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> While Divjot and others note the > > specialization for > > > > > > > > > > >> UTF-8, > > > > > > > > > > >>>>>> Parquet > > > > > > > > > > >>>>>>>>>> already > > > > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings > > > > > > > > > > >>>>>>>>>>> < > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/README.md#types > > > > > > > > > > >>>>>>>>>> . > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR > > would > > > > be a > > > > > > new > > > > > > > > > > >>>>>>> restriction > > > > > > > > > > >>>>>>>>> on > > > > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar > > > > > > FLOAT/DOUBLE > > > > > > > > > > >>>>> columns > > > > > > > > > > >>>>>>>> have > > > > > > > > > > >>>>>>>>>>> never had such a rule. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> It would also not match how the systems that > > > > actually > > > > > > > > > > >>> produce > > > > > > > > > > >>>>>> these > > > > > > > > > > >>>>>>>>>> vectors > > > > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, > > e.g. > > > > > Spark, > > > > > > > > > > >>>> DuckDB, > > > > > > > > > > >>>>>>>> NumPy, > > > > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, > > oneMKL, > > > > and > > > > > > > > > > >>>> OpenBLAS. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule > > is an > > > > > > > > > > >> insert > > > > > > > > > > >>>>>> policy. > > > > > > > > > > >>>>>>>>> While > > > > > > > > > > >>>>>>>>>> I > > > > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format > > should > > > > > allow > > > > > > > > > > >> them > > > > > > > > > > >>>> out > > > > > > > > > > >>>>>> of > > > > > > > > > > >>>>>>>> the > > > > > > > > > > >>>>>>>>>> box. > > > > > > > > > > >>>>>>>>>>> It is a user system decision to disallow > > writing > > > > > these > > > > > > > > > > >>> values > > > > > > > > > > >>>>> or > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>>>> throw > > > > > > > > > > >>>>>>>>>>> errors when encountering them. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN > > out of > > > > > > vector > > > > > > > > > > >>>> math. > > > > > > > > > > >>>>>>>> Finite > > > > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance: > > > > magnitude = > > > > > > > > > > >>>>>> ||[FLOAT_MAX, > > > > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite > > > > > overflows > > > > > > > > > > >> to > > > > > > > > > > >>>>> +Inf. > > > > > > > > > > >>>>>>>>> PyTorch > > > > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20, > > > > > > 1e20]).norm() > > > > > > > > > > >> is > > > > > > > > > > >>>> inf > > > > > > > > > > >>>>>> in > > > > > > > > > > >>>>>>>>> fp32. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then > > be Inf > > > > > > while > > > > > > > > > > >>> the > > > > > > > > > > >>>>>>> VECTOR > > > > > > > > > > >>>>>>>>>> itself > > > > > > > > > > >>>>>>>>>>> was required to be finite. The ban only > forbids > > > > > storing > > > > > > > > > > >>>>> specials > > > > > > > > > > >>>>>> in > > > > > > > > > > >>>>>>>> the > > > > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple > > > > > > downstream > > > > > > > > > > >>>>>>> computations > > > > > > > > > > >>>>>>>>>> from > > > > > > > > > > >>>>>>>>>>> producing them. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add > an > > > > > > optional > > > > > > > > > > >>> bool > > > > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines > that > > > > want > > > > > > > > > > >>>>>> pgvector-like > > > > > > > > > > >>>>>>>>>>> rejection can require the flag = true. > Engines > > that > > > > > use > > > > > > > > > > >>>> NumPy / > > > > > > > > > > >>>>>>>>> PyTorch / > > > > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE > > values. > > > > For > > > > > > the > > > > > > > > > > >>> same > > > > > > > > > > >>>>>>>> reason, I > > > > > > > > > > >>>>>>>>>> do > > > > > > > > > > >>>>>>>>>>> not favor adding yet another type called > > > > > FINITE_VECTOR. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>> Sebastian > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora > > via > > > > dev > > > > > < > > > > > > > > > > >>>>>>>>>>> [email protected]> > > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> I agree with moving forward with the > proposal > > and > > > > > > > > > > >>> renaming > > > > > > > > > > >>>>>> VECTOR > > > > > > > > > > >>>>>>>> to > > > > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something > > > > > > > > > > >>>>>>>>>>>> similar if others feel that better > > communicates > > > > its > > > > > > > > > > >>>> intent. I > > > > > > > > > > >>>>>> see > > > > > > > > > > >>>>>>>>> value > > > > > > > > > > >>>>>>>>>>> in > > > > > > > > > > >>>>>>>>>>>> having a vector type that guarantees > > > > > > > > > > >>>>>>>>>>>> its values are mathematically-correct > vectors > > for > > > > a > > > > > > few > > > > > > > > > > >>>>>> reasons: > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar > > to > > > > the > > > > > > > > > > >>> STRING > > > > > > > > > > >>>>>>> logical > > > > > > > > > > >>>>>>>>> type > > > > > > > > > > >>>>>>>>>>>> mandating UTF-8. > > > > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics > > for > > > > > > > > > > >> detecting > > > > > > > > > > >>>> NaN > > > > > > > > > > >>>>>> or > > > > > > > > > > >>>>>>>>> +-inf > > > > > > > > > > >>>>>>>>>>>> values as all statistics are optional. > > > > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for > > > > requiring > > > > > > > > > > >> that > > > > > > > > > > >>>>>> writers > > > > > > > > > > >>>>>>>>> emit > > > > > > > > > > >>>>>>>>>>>> stats for specific logical types. > > > > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could > > flow > > > > > > > > > > >> through > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>> engine > > > > > > > > > > >>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>> block them for specific functionality. > > > > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, > I > > > > don't > > > > > > > > > > >> think > > > > > > > > > > >>>>> this > > > > > > > > > > >>>>>> is > > > > > > > > > > >>>>>>>> as > > > > > > > > > > >>>>>>>>>> easy > > > > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run > > > > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for > > > > > examples). > > > > > > > > > > >>> Their > > > > > > > > > > >>>>>>> Python > > > > > > > > > > >>>>>>>>> [3] > > > > > > > > > > >>>>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on > > > > > > > > > > >>>>>>>>>>>> vectors that contain NaN values. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose > > format, > > > > > but > > > > > > I > > > > > > > > > > >>>> don't > > > > > > > > > > >>>>>>> think > > > > > > > > > > >>>>>>>>>> this > > > > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types. > > > > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where > > Parquet > > > > > > > > > > >> would > > > > > > > > > > >>> be > > > > > > > > > > >>>>>>>>> considered > > > > > > > > > > >>>>>>>>>> a > > > > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with > > > > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list > > type > > > > if > > > > > > we > > > > > > > > > > >>>>> believe > > > > > > > > > > >>>>>>>> there > > > > > > > > > > >>>>>>>>>> are > > > > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit > > > > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> [1] > > > > https://github.com/lance-format/lance/pull/3648 > > > > > > > > > > >>>>>>>>>>>> [2] > > > > > https://github.com/lance-format/lance/issues/8607 > > > > > > > > > > >>> (see > > > > > > > > > > >>>>>>> point 2 > > > > > > > > > > >>>>>>>>> in > > > > > > > > > > >>>>>>>>>>>> "root > > > > > > > > > > >>>>>>>>>>>> cause") > > > > > > > > > > >>>>>>>>>>>> [3] > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > > > > > > >>>>>>>>>>>> [4] > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp > > Fischbeck > > > > > via > > > > > > > > > > >>> dev < > > > > > > > > > > >>>>>>>>>>>> [email protected]> wrote: > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the > > proposal as > > > > > is. > > > > > > > > > > >> I > > > > > > > > > > >>>> see > > > > > > > > > > >>>>>>> strong > > > > > > > > > > >>>>>>>>>> value > > > > > > > > > > >>>>>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and > I > > > > agree > > > > > > > > > > >> with > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>>>>>> restrictions > > > > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in > > > > general. > > > > > > > > > > >>>>> Precedent > > > > > > > > > > >>>>>>>>> exists > > > > > > > > > > >>>>>>>>>> in > > > > > > > > > > >>>>>>>>>>>>> other types that restrict which physical > type > > > > > values > > > > > > > > > > >>> are > > > > > > > > > > >>>>>>>> logically > > > > > > > > > > >>>>>>>>>>> valid > > > > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be > > > > > > > > > > >> WKB-encoded). > > > > > > > > > > >>>> A > > > > > > > > > > >>>>>>>> general > > > > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think > > it > > > > > makes > > > > > > > > > > >>>> sense > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>>>>>> separate > > > > > > > > > > >>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and > I > > can > > > > > see > > > > > > > > > > >>> how > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>> two > > > > > > > > > > >>>>>>>>>> types > > > > > > > > > > >>>>>>>>>>>>> might diverge further in the future > regarding > > > > > > > > > > >>> statistics > > > > > > > > > > >>>> or > > > > > > > > > > >>>>>>>>>> encodings. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector > > type, > > > > but > > > > > it > > > > > > > > > > >>>> seems > > > > > > > > > > >>>>>>> like > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>> VECTOR > > > > > > > > > > >>>>>>>>>>>>> name is already used across other engines > > > > mentioned > > > > > > > > > > >> in > > > > > > > > > > >>>> this > > > > > > > > > > >>>>>>>> thread. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>>> Philipp > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu < > > > > > > > > > > >>>> [email protected]> > > > > > > > > > > >>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we > > > > should > > > > > > > > > > >>> treat > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>>>> general > > > > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical > > layout, > > > > > > > > > > >> and > > > > > > > > > > >>>>>> separate > > > > > > > > > > >>>>>>>> it > > > > > > > > > > >>>>>>>>>> from > > > > > > > > > > >>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or > > broader > > > > ML > > > > > > > > > > >>>> tensor) > > > > > > > > > > >>>>>>>> logical > > > > > > > > > > >>>>>>>>>>> type. > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to > define a > > > > > > > > > > >>>> specialized > > > > > > > > > > >>>>>>> VECTOR > > > > > > > > > > >>>>>>>>>> type > > > > > > > > > > >>>>>>>>>>> so > > > > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a > > common > > > > > > > > > > >>>> semantic > > > > > > > > > > >>>>>>>>> contract > > > > > > > > > > >>>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>> interoperability. > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just > the > > > > > > > > > > >>>> NaN/Inf/null > > > > > > > > > > >>>>>>>>>> limitations, > > > > > > > > > > >>>>>>>>>>>>> maybe > > > > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a > > qualification to > > > > > > > > > > >> the > > > > > > > > > > >>>> name > > > > > > > > > > >>>>>>> (like > > > > > > > > > > >>>>>>>>>>> Andrew > > > > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or > > > > > FINITE_VECTOR) > > > > > > > > > > >>> to > > > > > > > > > > >>>>>> avoid > > > > > > > > > > >>>>>>>>>>> confusion > > > > > > > > > > >>>>>>>>>>>>> with > > > > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we > > could > > > > > > > > > > >>>> introduce a > > > > > > > > > > >>>>>>>> boolean > > > > > > > > > > >>>>>>>>>>> flag > > > > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as > > > > > proposed > > > > > > > > > > >>>>>> earlier. > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew > Lamb > > < > > > > > > > > > > >>>>>>>>>> [email protected]> > > > > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points > of > > > > > > > > > > >>>> disagreement > > > > > > > > > > >>>>>>> here. > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too > > general for > > > > > > > > > > >> the > > > > > > > > > > >>>>>>> specific > > > > > > > > > > >>>>>>>>> use > > > > > > > > > > >>>>>>>>>>> case > > > > > > > > > > >>>>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite > values > > used > > > > > > > > > > >> for > > > > > > > > > > >>>>>> vector > > > > > > > > > > >>>>>>>>>>>> embeddings? > > > > > > > > > > >>>>>>>>>>>>>> If > > > > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to > > the > > > > > > > > > > >> name > > > > > > > > > > >>>>>>>>>> (FINITE_VECTOR? > > > > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for > > defining > > > > > > > > > > >>> logical > > > > > > > > > > >>>>>> types > > > > > > > > > > >>>>>>>>> that > > > > > > > > > > >>>>>>>>>>>>> specify > > > > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their > > > > > > > > > > >> contents > > > > > > > > > > >>> to > > > > > > > > > > >>>>>> help > > > > > > > > > > >>>>>>>>>> readers > > > > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] > or > > > > JSON; > > > > > > > > > > >>>>>> Defining > > > > > > > > > > >>>>>>>> one > > > > > > > > > > >>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>> vector > > > > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me. > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> Andrew > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> [1]: > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel > > Weeks < > > > > > > > > > > >>>>>>>>> [email protected]> > > > > > > > > > > >>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose > > format. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever > policy > > fits > > > > > > > > > > >>>> their > > > > > > > > > > >>>>>>>> usage, > > > > > > > > > > >>>>>>>>>> but > > > > > > > > > > >>>>>>>>>>>>> we're > > > > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file > > format > > > > > > > > > > >>> that > > > > > > > > > > >>>>> aims > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>> be > > > > > > > > > > >>>>>>>>>>>> broadly > > > > > > > > > > >>>>>>>>>>>>>>>>> applicable. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to > be > > > > > > > > > > >> general > > > > > > > > > > >>>>>>> purpose, > > > > > > > > > > >>>>>>>>> but > > > > > > > > > > >>>>>>>>>>>> types > > > > > > > > > > >>>>>>>>>>>>>>> within > > > > > > > > > > >>>>>>>>>>>>>>>> the format define specific > representation > > and > > > > > > > > > > >>>>> semantics > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>>> achieve > > > > > > > > > > >>>>>>>>>>>>> that > > > > > > > > > > >>>>>>>>>>>>>>>> goal. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than > > ML and > > > > > > > > > > >>>> it's > > > > > > > > > > >>>>>>> quite > > > > > > > > > > >>>>>>>>>>> trivial > > > > > > > > > > >>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values > > using > > > > > > > > > > >>> e.g. > > > > > > > > > > >>>>>> NumPy > > > > > > > > > > >>>>>>>> or > > > > > > > > > > >>>>>>>>>>> Pandas > > > > > > > > > > >>>>>>>>>>>>> (or > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or > > scientific > > > > > > > > > > >>>>> computing > > > > > > > > > > >>>>>>>>>> software). > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting > > non-finite > > > > > > > > > > >>>> values > > > > > > > > > > >>>>> is > > > > > > > > > > >>>>>>>>>> directly > > > > > > > > > > >>>>>>>>>>>>>> related > > > > > > > > > > >>>>>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector. > Vectors > > > > have > > > > > > > > > > >>>>>> specific > > > > > > > > > > >>>>>>>>>>>>>> characteristics > > > > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc). > > > > > > > > > > >>> Non-finite > > > > > > > > > > >>>>>>> values > > > > > > > > > > >>>>>>>>> are > > > > > > > > > > >>>>>>>>>>>>> invalid > > > > > > > > > > >>>>>>>>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>>>>>> break those characteristics. I'd liken > > it to > > > > > > > > > > >>>> storing > > > > > > > > > > >>>>>>>>> non-utf8 > > > > > > > > > > >>>>>>>>>>>> values > > > > > > > > > > >>>>>>>>>>>>>> in > > > > > > > > > > >>>>>>>>>>>>>>> a > > > > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a > > fixed > > > > > > > > > > >>>> length > > > > > > > > > > >>>>>>> list, > > > > > > > > > > >>>>>>>>>> which > > > > > > > > > > >>>>>>>>>>>> is > > > > > > > > > > >>>>>>>>>>>>>> more > > > > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same > > semantic > > > > > > > > > > >>>> meaning. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all > > > > > > > > > > >>> similarly > > > > > > > > > > >>>>>> shaped > > > > > > > > > > >>>>>>>> use > > > > > > > > > > >>>>>>>>>>> cases > > > > > > > > > > >>>>>>>>>>>>>> into > > > > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size > list). > > > > > > > > > > >> Vector > > > > > > > > > > >>>> is > > > > > > > > > > >>>>> a > > > > > > > > > > >>>>>>> well > > > > > > > > > > >>>>>>>>>>>>> established > > > > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with > > the > > > > > > > > > > >>> general > > > > > > > > > > >>>>>>> purpose > > > > > > > > > > >>>>>>>>>>> nature > > > > > > > > > > >>>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases > > > > > > > > > > >> natively > > > > > > > > > > >>>>>> support > > > > > > > > > > >>>>>>>>>> this). > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> -Dan > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine > > > > > > > > > > >> Pitrou < > > > > > > > > > > >>>>>>>>>>>> [email protected]> > > > > > > > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer > a > > > > > > > > > > >> écrit > > > > > > > > > > >>> : > > > > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet > is > > the > > > > > > > > > > >>>> clear > > > > > > > > > > >>>>>>>>> use-case > > > > > > > > > > >>>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>>>> non-finite > > > > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has > > > > > > > > > > >> been a > > > > > > > > > > >>>> lot > > > > > > > > > > >>>>>> of > > > > > > > > > > >>>>>>>>>>>> theoretical > > > > > > > > > > >>>>>>>>>>>>>>>>> discussion > > > > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings > > > > > > > > > > >>>> currently > > > > > > > > > > >>>>>>>> produce > > > > > > > > > > >>>>>>>>>>> these > > > > > > > > > > >>>>>>>>>>>>>>> vectors > > > > > > > > > > >>>>>>>>>>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream > > engines? > > > > > > > > > > >>> I'd > > > > > > > > > > >>>>>> like > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>> limit > > > > > > > > > > >>>>>>>>>>>>>>>> introducing > > > > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values > > > > > > > > > > >> which > > > > > > > > > > >>>> are > > > > > > > > > > >>>>>>> simply > > > > > > > > > > >>>>>>>>> are > > > > > > > > > > >>>>>>>>>>>>> failure > > > > > > > > > > >>>>>>>>>>>>>>>> modes > > > > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than > > ML and > > > > > > > > > > >>>> it's > > > > > > > > > > >>>>>>> quite > > > > > > > > > > >>>>>>>>>>> trivial > > > > > > > > > > >>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values > > using > > > > > > > > > > >>> e.g. > > > > > > > > > > >>>>>> NumPy > > > > > > > > > > >>>>>>>> or > > > > > > > > > > >>>>>>>>>>> Pandas > > > > > > > > > > >>>>>>>>>>>>> (or > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or > > scientific > > > > > > > > > > >>>>> computing > > > > > > > > > > >>>>>>>>>> software). > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a > > vector > > > > > > > > > > >>> is > > > > > > > > > > >>>> a > > > > > > > > > > >>>>>> IETF > > > > > > > > > > >>>>>>>>>>> standard > > > > > > > > > > >>>>>>>>>>>> or > > > > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or > UTF8 > > > > > > > > > > >> (where > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>>> values > > > > > > > > > > >>>>>>>>>> are > > > > > > > > > > >>>>>>>>>>>>>> expected > > > > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective > > > > > > > > > > >> specs) > > > > > > > > > > >>> do > > > > > > > > > > >>>>> not > > > > > > > > > > >>>>>>>>> apply. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a > better > > > > > > > > > > >> chance > > > > > > > > > > >>>> of > > > > > > > > > > >>>>>>>>> converging > > > > > > > > > > >>>>>>>>>>> if > > > > > > > > > > >>>>>>>>>>>> we > > > > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns > of: > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length > list/vector > > data > > > > > > > > > > >>> in > > > > > > > > > > >>>>>>> Parquet > > > > > > > > > > >>>>>>>>>> (with > > > > > > > > > > >>>>>>>>>>> a > > > > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested > types) > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a > > given > > > > > > > > > > >>>> column > > > > > > > > > > >>>>>> (or > > > > > > > > > > >>>>>>>>>> column > > > > > > > > > > >>>>>>>>>>>>> chunk, > > > > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite > values > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their > own > > > > > > > > > > >>> right, > > > > > > > > > > >>>>> but > > > > > > > > > > >>>>>> do > > > > > > > > > > >>>>>>>> not > > > > > > > > > > >>>>>>>>>>> have > > > > > > > > > > >>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>> be > > > > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Regards > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Antoine. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
