> Yes, I expect different functions in engines. I think pgvector uses more > specific functions for vectors, like `vector_dims` and `vector_norms`. As > far as I'm aware, these aren't available for arrays without an explicit > cast.
pg_vector is IMHO just one example that cannot be generalized. The reason (I suspect) that it uses its own type is that PostgreSQL's array type is somewhat overly bloated. It is not fixed length and it even supports multiple dimensions, leading to an overly verbose binary format. Had PostgreSQL a fixed-length array type that was just represented as the values in binary, I think that pgvector would not have chosen an own type. In contrast, I would expect most engines that already have a fixed-length array type to just use that for vectors. Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <[email protected]>: > > Could the proposal include a compatibility section addressing unannotated > list data before we vote? > > Will, would a paragraph like this [1] be what you have in mind? > > On the topic of the vote - I would like to call it later today based on > parquet-format [2]. > > [1] > > https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4 > [2] https://github.com/apache/parquet-format/pull/624 > > Best, > Rok > > On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev < > [email protected]> > wrote: > > > Thanks Ryan, Dan. > > > > My questions about what the type enables, and how you think Iceberg > tooling > > would use it, stem from the backward-compatibility concerns I've raised > > earlier in the thread. I was trying to weigh the cost against the > benefit. > > I worry that this annotation will fragment and orphan a lot of data: > > perfectly good vectors that become second-class just because they weren't > > written with the annotation. > > > > There is an enormous amount of embedding data in Parquet today. Many many > > lottabytes of it :) Some of it is packed in bespoke formats, but a very > > large share is plain, unannotated lists of floats or doubles. In addition > > to all this existing data, many pipelines will keep writing new data this > > way too, because they are unaware of the new annotation or can't be > > changed. > > > > Ryan, you said compatibility isn't a concern because this is a new type. > I > > think that assumes the data is new data written by an Iceberg writer. But > > many Iceberg tables are built from files written by other engines. The > > standard implementations support this directly (add_files, migrate, name > > mapping), and the popular public embedding datasets are a typical source. > > If Iceberg requires vector columns to carry the annotation, none of those > > files can back a vector column without being rewritten, even though the > > bytes would be identical. > > > > It makes sense for Iceberg writers to check for inf/nan/null when they > > write a vector column. But since many files come from other writers, > robust > > readers should check correctness anyway, as they do with UTF-8 for > strings. > > > > So my position remains that this would be better done entirely in Iceberg > > rather than pushed down to Parquet. Iceberg ought to accept unannotated > > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via > add_files > > and a metadata-only list → vector promotion. > > > > Could the proposal include a compatibility section addressing unannotated > > list data before we vote? > > > > Will > > > > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <[email protected]> > > wrote: > > > > > > 1. do you expect that the backing files that are parquet MUST use the > > new > > > logical vector type? > > > > > > I expect that the Iceberg spec will require writers to produce the > > > annotation. > > > > > > > 2. do you expect the software accessing an iceberg table . . . to > > offer > > > different functions on vector columns compared to naked lists? > > > > > > Yes, I expect different functions in engines. I think pgvector uses > more > > > specific functions for vectors, like `vector_dims` and `vector_norms`. > As > > > far as I'm aware, these aren't available for arrays without an explicit > > > cast. > > > > > > > Will a nearest neighbour search on a naked list of doubles be > > supported, > > > for example? > > > > > > This is an engine decision, but a list like this would not be exposed > as > > a > > > vector. You would most likely cast it to a vector in order to call > vector > > > functions. > > > > > > > 3. And will non-parquet backing files like avro and orc be supported > > for > > > the > > > vector columns? > > > > > > All Iceberg types can be stored in supported file formats. > > > > > > It seems like the direction for this line of questioning is basically > the > > > same as before: Can we just treat list<float> as a synonym of vector? > > > > > > And again, I think this is the wrong question. We have users right now > > that > > > struggle with errors that are found at read time, like the dimensions > of > > > their vectors not matching. Finding this out at read time is a problem > > and > > > we want to shift validation to write time so we can catch errors > earlier > > > and have guarantees that the data is usable. As I mentioned, one of the > > > ways we have customers using data is sharing it with systems like > > pgvector > > > that also have these requirements. Just because you can think of a > vector > > > as a very special list of floats doesn't mean that write time > guarantees > > > are unreasonable or unnecessary. > > > > > > Ryan > > > > > > > > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < > > > [email protected]> wrote: > > > > > > > Thx for replying. > > > > > > > > I keep trying to explain that there are no performance advantages. > But > > > > anyway :) > > > > > > > > I understand you are envisaging a vector column type (with the fixed > > > length > > > > nan inf constraints you’ve been detailing) being added to iceberg? > > Then > > > > the following choices are interesting: > > > > > > > > 1. do you expect that the backing files that are parquet MUST use the > > new > > > > logical vector type? > > > > > > > > 2. do you expect the software accessing an iceberg table - sparksql, > > for > > > > example - to offer different functions on vector columns compared to > > > naked > > > > lists? Will a nearest neighbour search on a naked list of doubles be > > > > supported, for example? > > > > > > > > 3. And will non-parquet backing files like avro and orc be supported > > for > > > > the vector columns? > > > > > > > > thx > > > > > > > > Will > > > > > > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > > > > > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now > and > > > > would > > > > work with VECTOR. Jan asked the same question, and I don't think > either > > > of > > > > us has seen one yet. > > > > > > > > I think this is the wrong question to ask, but I'm glad that you're > > > asking > > > > directly. Clearly, a variable-length list is a superset of a > > fixed-length > > > > list. And a list of floats is a superset of a list of finite floats. > A > > > list > > > > of optional elements is a superset of a list of required elements. I > > > don't > > > > think there's a use case that _can't_ be done. > > > > > > > > Is there a use case that can't be done with binary in place of > string? > > > And > > > > why don't we use JSON or Variant for everything? Just because we can > > use > > > a > > > > wider type doesn't mean that we don't benefit from communicating > > > guarantees > > > > like "this must be UTF-8". > > > > > > > > Types introduce guarantees that we can use to get better performance > > and > > > to > > > > either prevent errors or raise them closer to where data is produced. > > > > > > > > For vectors, the guarantee that elements are finite gives us better > > > > performance because we don't need to check each element. This also > > moves > > > > the error closer to where data is produced because writers are > required > > > to > > > > verify the guarantees about each element and will detect bad values > at > > > > write time rather than at read time. > > > > > > > > Ryan > > > > > > > > > >
