Hi Rok Thanks for continuing to push this proposal forward.
> 1. Are the proposed level and null-vector semantics appropriate? This proposal makes sense to me. One thing I'm not sure about is the use of padded definition levels. This could be required to support element nullability cleanly with something like the three-level schema structure of the existing list logical type. But maybe only one definition level per vector should be used if the child schema nodes all contribute no definition levels? > 2. What nesting should be supported initially? In theory, it seems like it should be possible to nest a vector type within another repeated type, or have a repeated type within a vector type. But I can imagine that handling these scenarios would get quite complex. Only supporting a single level of nesting within an optional group like in your embedding example might be a reasonable limitation to start with, and would likely cover the majority of use cases. > 3. How should num_values, page indexes, offset indexes, and null counts be > defined? num_values is defined in both ColumnMetaData and the data page headers. I think in both cases, num_values should be the total number of element values, for consistency with repetition=repeated. Then for consistency with num_values, a null vector should probably contribute vector_length values to the null_count. > 4. How should unsupported readers detect and reject this representation? I'd hope that the presence of an unrecognized repetition type would cause any existing library to return an error, but this is something that should be verified. Cheers, Adam On Tue, 28 Jul 2026 at 01:32, Rok Mihevc <[email protected]> wrote: > Hi all, > > This thread proposes adding a new VECTOR repetition level as the long-term > physical representation for fixed-size-lists. > > The existing thread has moved toward optimizing readers and writers for > fixed-size-list data stored using LIST. Those optimizations provide a good > compatible short-term path, but LIST still models variable-size-list data, > writes redundant repetition levels, and imposes unnecessary writer and > reader complexity and storage overhead. > > I propose extending the schema as follows: > > ```thrift > enum FieldRepetitionType { > REQUIRED = 0, > OPTIONAL = 1, > REPEATED = 2, > VECTOR = 3 > } > > struct SchemaElement { > // Existing fields... > > // Required when repetition_type == VECTOR > optional i32 vector_length; > } > ``` > > VECTOR would mean that a field occurs exactly vector_length times for each > parent value. Unlike REPEATED, the VECTOR dimension would not increase > either the maximum repetition level or the maximum definition > level. > > For example, a nullable 768-dimensional float vector could be represented > as: > > ```text > optional group embedding { > vector float element [768]; > } > ``` > > The optional group controls the nullability of the vector as a whole. The > VECTOR child describes 768 dense, non-null float elements. > > The main properties of this representation would be: > > - The VECTOR dimension produces no repetition levels. > - Elements remain a normal Parquet column, so existing encodings such as > PLAIN, BYTE_STREAM_SPLIT, dictionary encoding, and ALP can operate on the > element type. > - Every present vector contains exactly vector_length contiguous elements. > - A vector must not be split across data pages. > - Nullable vectors use the existing definition-level mechanism. I propose > padded definition-level semantics so that each parent row still occupies > vector_length positions. > - For an initial version, VECTOR elements would be non-null. VECTOR nesting > could be supported, while variable-length repeated descendants would be > excluded. > - Statistics remain element-level; vector-level ordering and min/max > statistics would not be defined. > > This is intentionally a breaking format addition. Readers that do not > recognize VECTOR cannot safely interpret the column and must reject it. We > therefore need an explicit compatibility or versioning story. > > Compared with an annotated LIST, this representation removes redundant > levels instead of merely allowing optimized readers to ignore them. It also > preserves element-wise encodings and avoids retaining variable-length LIST > semantics as part of the long-term vector representation. > > The VECTOR logical type is a related but separate concern. It could > annotate both today’s LIST representation and this native representation, > allowing applications to identify vector semantics independently of how the > values are physically serialized. > > The main questions I would like feedback on are: > > 1. Are the proposed level and null-vector semantics appropriate? > 2. What nesting should be supported initially? > 3. How should num_values, page indexes, offset indexes, and null counts be > defined? > 4. How should unsupported readers detect and reject this representation? > > Relevant material: > > - Existing discussion: > https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3 > - Design document: > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit > - Format prototype: https://github.com/apache/parquet-format/pull/592 > - Arrow Go prototype: https://github.com/apache/arrow-go/pull/854 > > Best, > Rok >
