Thanks Ryan, Dan. My questions about what the type enables, and how you think Iceberg tooling would use it, stem from the backward-compatibility concerns I've raised earlier in the thread. I was trying to weigh the cost against the benefit. I worry that this annotation will fragment and orphan a lot of data: perfectly good vectors that become second-class just because they weren't written with the annotation.
There is an enormous amount of embedding data in Parquet today. Many many lottabytes of it :) Some of it is packed in bespoke formats, but a very large share is plain, unannotated lists of floats or doubles. In addition to all this existing data, many pipelines will keep writing new data this way too, because they are unaware of the new annotation or can't be changed. Ryan, you said compatibility isn't a concern because this is a new type. I think that assumes the data is new data written by an Iceberg writer. But many Iceberg tables are built from files written by other engines. The standard implementations support this directly (add_files, migrate, name mapping), and the popular public embedding datasets are a typical source. If Iceberg requires vector columns to carry the annotation, none of those files can back a vector column without being rewritten, even though the bytes would be identical. It makes sense for Iceberg writers to check for inf/nan/null when they write a vector column. But since many files come from other writers, robust readers should check correctness anyway, as they do with UTF-8 for strings. So my position remains that this would be better done entirely in Iceberg rather than pushed down to Parquet. Iceberg ought to accept unannotated LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via add_files and a metadata-only list → vector promotion. Could the proposal include a compatibility section addressing unannotated list data before we vote? Will On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <[email protected]> wrote: > > 1. do you expect that the backing files that are parquet MUST use the new > logical vector type? > > I expect that the Iceberg spec will require writers to produce the > annotation. > > > 2. do you expect the software accessing an iceberg table . . . to offer > different functions on vector columns compared to naked lists? > > Yes, I expect different functions in engines. I think pgvector uses more > specific functions for vectors, like `vector_dims` and `vector_norms`. As > far as I'm aware, these aren't available for arrays without an explicit > cast. > > > Will a nearest neighbour search on a naked list of doubles be supported, > for example? > > This is an engine decision, but a list like this would not be exposed as a > vector. You would most likely cast it to a vector in order to call vector > functions. > > > 3. And will non-parquet backing files like avro and orc be supported for > the > vector columns? > > All Iceberg types can be stored in supported file formats. > > It seems like the direction for this line of questioning is basically the > same as before: Can we just treat list<float> as a synonym of vector? > > And again, I think this is the wrong question. We have users right now that > struggle with errors that are found at read time, like the dimensions of > their vectors not matching. Finding this out at read time is a problem and > we want to shift validation to write time so we can catch errors earlier > and have guarantees that the data is usable. As I mentioned, one of the > ways we have customers using data is sharing it with systems like pgvector > that also have these requirements. Just because you can think of a vector > as a very special list of floats doesn't mean that write time guarantees > are unreasonable or unnecessary. > > Ryan > > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < > [email protected]> wrote: > > > Thx for replying. > > > > I keep trying to explain that there are no performance advantages. But > > anyway :) > > > > I understand you are envisaging a vector column type (with the fixed > length > > nan inf constraints you’ve been detailing) being added to iceberg? Then > > the following choices are interesting: > > > > 1. do you expect that the backing files that are parquet MUST use the new > > logical vector type? > > > > 2. do you expect the software accessing an iceberg table - sparksql, for > > example - to offer different functions on vector columns compared to > naked > > lists? Will a nearest neighbour search on a naked list of doubles be > > supported, for example? > > > > 3. And will non-parquet backing files like avro and orc be supported for > > the vector columns? > > > > thx > > > > Will > > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and > > would > > work with VECTOR. Jan asked the same question, and I don't think either > of > > us has seen one yet. > > > > I think this is the wrong question to ask, but I'm glad that you're > asking > > directly. Clearly, a variable-length list is a superset of a fixed-length > > list. And a list of floats is a superset of a list of finite floats. A > list > > of optional elements is a superset of a list of required elements. I > don't > > think there's a use case that _can't_ be done. > > > > Is there a use case that can't be done with binary in place of string? > And > > why don't we use JSON or Variant for everything? Just because we can use > a > > wider type doesn't mean that we don't benefit from communicating > guarantees > > like "this must be UTF-8". > > > > Types introduce guarantees that we can use to get better performance and > to > > either prevent errors or raise them closer to where data is produced. > > > > For vectors, the guarantee that elements are finite gives us better > > performance because we don't need to check each element. This also moves > > the error closer to where data is produced because writers are required > to > > verify the guarantees about each element and will detect bad values at > > write time rather than at read time. > > > > Ryan > > >
