I still believe that FINITE_VECTOR, as proposed in PR #624 <https://github.com/apache/parquet-format/pull/624>, is not the right abstraction for a general-purpose format because it lacks composability.
I've drafted an alternative proposal (PR #633) <https://github.com/apache/parquet-format/pull/633> to demonstrate how we could achieve the same functionality in a more composable way. This is by no means ready to merge; it's primarily intended to illustrate the approach. The idea is simple: introduce two orthogonal building blocks: - *Fixed-size lists:* Give lists an optional elementCount. - *Finite floats:* Give floating-point types an optional isFinite property. These two additions would allow us to express FINITE_VECTOR, but also enable a much broader range of useful types: - Fixed-size lists of *any* element type, not just those supported by FINITE_VECTOR. - Finite floats in any context, whether standalone, in fixed- or variable-size lists, in maps, or elsewhere. - Higher order tensors. You can just have list<list<finite float, 64>, 64>> for example. As a bonus, fixed-size lists would remain readable by legacy readers, which would simply interpret them as variable-size lists. As finiteness is applied only to floats, we also don't get the semantic weirdness where FINITE_VECTOR can also annotate a vector of bools or ints, which are trivially always finite. My main concern with FINITE_VECTOR is that it's highly specific to embedding vectors. While embeddings are certainly a hot topic today, I think baking such feature-specific abstractions into a general-purpose format intended to remain relevant for decades is the wrong trade-off. Different systems have different requirements, and those requirements will evolve. I'd much rather see us build a small set of *orthogonal, composable types* from which specialized types like FINITE_VECTOR can be constructed, rather than introducing those specialized types directly into the format. Am Mo., 2. Nov. 2026 um 20:23 Uhr schrieb Jan Finis <[email protected]>: > > Yes, I expect different functions in engines. I think pgvector uses more > > specific functions for vectors, like `vector_dims` and `vector_norms`. As > > far as I'm aware, these aren't available for arrays without an explicit > > cast. > > pg_vector is IMHO just one example that cannot be generalized. The reason > (I suspect) that it uses its own type is that PostgreSQL's array type is > somewhat > overly bloated. It is not fixed length and it even supports multiple > dimensions, leading > to an overly verbose binary format. Had PostgreSQL a fixed-length array > type that > was just represented as the values in binary, I think that pgvector would > not have > chosen an own type. > > In contrast, I would expect most engines that already have a fixed-length > array type > to just use that for vectors. > > Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <[email protected] > >: > >> > Could the proposal include a compatibility section addressing >> unannotated >> list data before we vote? >> >> Will, would a paragraph like this [1] be what you have in mind? >> >> On the topic of the vote - I would like to call it later today based on >> parquet-format [2]. >> >> [1] >> >> https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4 >> [2] https://github.com/apache/parquet-format/pull/624 >> >> Best, >> Rok >> >> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev < >> [email protected]> >> wrote: >> >> > Thanks Ryan, Dan. >> > >> > My questions about what the type enables, and how you think Iceberg >> tooling >> > would use it, stem from the backward-compatibility concerns I've raised >> > earlier in the thread. I was trying to weigh the cost against the >> benefit. >> > I worry that this annotation will fragment and orphan a lot of data: >> > perfectly good vectors that become second-class just because they >> weren't >> > written with the annotation. >> > >> > There is an enormous amount of embedding data in Parquet today. Many >> many >> > lottabytes of it :) Some of it is packed in bespoke formats, but a very >> > large share is plain, unannotated lists of floats or doubles. In >> addition >> > to all this existing data, many pipelines will keep writing new data >> this >> > way too, because they are unaware of the new annotation or can't be >> > changed. >> > >> > Ryan, you said compatibility isn't a concern because this is a new >> type. I >> > think that assumes the data is new data written by an Iceberg writer. >> But >> > many Iceberg tables are built from files written by other engines. The >> > standard implementations support this directly (add_files, migrate, name >> > mapping), and the popular public embedding datasets are a typical >> source. >> > If Iceberg requires vector columns to carry the annotation, none of >> those >> > files can back a vector column without being rewritten, even though the >> > bytes would be identical. >> > >> > It makes sense for Iceberg writers to check for inf/nan/null when they >> > write a vector column. But since many files come from other writers, >> robust >> > readers should check correctness anyway, as they do with UTF-8 for >> strings. >> > >> > So my position remains that this would be better done entirely in >> Iceberg >> > rather than pushed down to Parquet. Iceberg ought to accept unannotated >> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via >> add_files >> > and a metadata-only list → vector promotion. >> > >> > Could the proposal include a compatibility section addressing >> unannotated >> > list data before we vote? >> > >> > Will >> > >> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <[email protected] >> > >> > wrote: >> > >> > > > 1. do you expect that the backing files that are parquet MUST use >> the >> > new >> > > logical vector type? >> > > >> > > I expect that the Iceberg spec will require writers to produce the >> > > annotation. >> > > >> > > > 2. do you expect the software accessing an iceberg table . . . to >> > offer >> > > different functions on vector columns compared to naked lists? >> > > >> > > Yes, I expect different functions in engines. I think pgvector uses >> more >> > > specific functions for vectors, like `vector_dims` and >> `vector_norms`. As >> > > far as I'm aware, these aren't available for arrays without an >> explicit >> > > cast. >> > > >> > > > Will a nearest neighbour search on a naked list of doubles be >> > supported, >> > > for example? >> > > >> > > This is an engine decision, but a list like this would not be exposed >> as >> > a >> > > vector. You would most likely cast it to a vector in order to call >> vector >> > > functions. >> > > >> > > > 3. And will non-parquet backing files like avro and orc be supported >> > for >> > > the >> > > vector columns? >> > > >> > > All Iceberg types can be stored in supported file formats. >> > > >> > > It seems like the direction for this line of questioning is basically >> the >> > > same as before: Can we just treat list<float> as a synonym of vector? >> > > >> > > And again, I think this is the wrong question. We have users right now >> > that >> > > struggle with errors that are found at read time, like the dimensions >> of >> > > their vectors not matching. Finding this out at read time is a problem >> > and >> > > we want to shift validation to write time so we can catch errors >> earlier >> > > and have guarantees that the data is usable. As I mentioned, one of >> the >> > > ways we have customers using data is sharing it with systems like >> > pgvector >> > > that also have these requirements. Just because you can think of a >> vector >> > > as a very special list of floats doesn't mean that write time >> guarantees >> > > are unreasonable or unnecessary. >> > > >> > > Ryan >> > > >> > > >> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < >> > > [email protected]> wrote: >> > > >> > > > Thx for replying. >> > > > >> > > > I keep trying to explain that there are no performance advantages. >> But >> > > > anyway :) >> > > > >> > > > I understand you are envisaging a vector column type (with the fixed >> > > length >> > > > nan inf constraints you’ve been detailing) being added to iceberg? >> > Then >> > > > the following choices are interesting: >> > > > >> > > > 1. do you expect that the backing files that are parquet MUST use >> the >> > new >> > > > logical vector type? >> > > > >> > > > 2. do you expect the software accessing an iceberg table - >> sparksql, >> > for >> > > > example - to offer different functions on vector columns compared to >> > > naked >> > > > lists? Will a nearest neighbour search on a naked list of doubles >> be >> > > > supported, for example? >> > > > >> > > > 3. And will non-parquet backing files like avro and orc be supported >> > for >> > > > the vector columns? >> > > > >> > > > thx >> > > > >> > > > Will >> > > > >> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: >> > > > >> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now >> and >> > > > would >> > > > work with VECTOR. Jan asked the same question, and I don't think >> either >> > > of >> > > > us has seen one yet. >> > > > >> > > > I think this is the wrong question to ask, but I'm glad that you're >> > > asking >> > > > directly. Clearly, a variable-length list is a superset of a >> > fixed-length >> > > > list. And a list of floats is a superset of a list of finite >> floats. A >> > > list >> > > > of optional elements is a superset of a list of required elements. I >> > > don't >> > > > think there's a use case that _can't_ be done. >> > > > >> > > > Is there a use case that can't be done with binary in place of >> string? >> > > And >> > > > why don't we use JSON or Variant for everything? Just because we can >> > use >> > > a >> > > > wider type doesn't mean that we don't benefit from communicating >> > > guarantees >> > > > like "this must be UTF-8". >> > > > >> > > > Types introduce guarantees that we can use to get better performance >> > and >> > > to >> > > > either prevent errors or raise them closer to where data is >> produced. >> > > > >> > > > For vectors, the guarantee that elements are finite gives us better >> > > > performance because we don't need to check each element. This also >> > moves >> > > > the error closer to where data is produced because writers are >> required >> > > to >> > > > verify the guarantees about each element and will detect bad values >> at >> > > > write time rather than at read time. >> > > > >> > > > Ryan >> > > > >> > > >> > >> >
