Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
Hi all, As proposed during the general community call we held a focused call today to discuss the proposal for adding a logical vector type to parquet. This was called because it seems: * Vectors (as in vector DB) storage needs are different to what we need for fixed size list applications we've been disucssing so far [0].
This seems unsubstantiated. Can you please provide an explanation? Ideally we'd use the *same* underlying solution for both problems.
We have converged on two options that could work, but have not yet decided which one to propose: - Option A - FLBA as physical storage (FLBA(num_elements * element_width)). Vector logical type would provide an enum specifying the element type, and the element would be non-nullable. The element type enum could be independent of the parquet type system, making evolution easier. Downside is loss of element-level encodings available today, though they can be added later. Upside is simplicity and ease of evolution. - Option C1 - LIST with numeric parquet type elements as physical storage. New float types (e.g. BF16) can be added in parquet proper as needed. Downside is we need to add new types to parquet, DL/RL overhead is likely not a concern. Upside is we have encoding support now.
Option A is a short-term solution with severe limitations (lack of encoding choices) that make me entirely negative about the idea.
I'm lukewarm about Option C (especially the optimistic statement that "DL/RL overhead is likely not a concern"), but at least it doesn't paint us in a corner.
Regards Antoine.
