Thx for replying. I keep trying to explain that there are no performance advantages. But anyway :)
I understand you are envisaging a vector column type (with the fixed length nan inf constraints you’ve been detailing) being added to iceberg? Then the following choices are interesting: 1. do you expect that the backing files that are parquet MUST use the new logical vector type? 2. do you expect the software accessing an iceberg table - sparksql, for example - to offer different functions on vector columns compared to naked lists? Will a nearest neighbour search on a naked list of doubles be supported, for example? 3. And will non-parquet backing files like avro and orc be supported for the vector columns? thx Will On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > I'm asking for a workload that can't be done on LIST<FLOAT> now and would work with VECTOR. Jan asked the same question, and I don't think either of us has seen one yet. I think this is the wrong question to ask, but I'm glad that you're asking directly. Clearly, a variable-length list is a superset of a fixed-length list. And a list of floats is a superset of a list of finite floats. A list of optional elements is a superset of a list of required elements. I don't think there's a use case that _can't_ be done. Is there a use case that can't be done with binary in place of string? And why don't we use JSON or Variant for everything? Just because we can use a wider type doesn't mean that we don't benefit from communicating guarantees like "this must be UTF-8". Types introduce guarantees that we can use to get better performance and to either prevent errors or raise them closer to where data is produced. For vectors, the guarantee that elements are finite gives us better performance because we don't need to check each element. This also moves the error closer to where data is produced because writers are required to verify the guarantees about each element and will detect bad values at write time rather than at read time. Ryan
