Hi all,
This thread proposes adding a new VECTOR repetition level as the long-term
physical representation for fixed-size-lists.
The existing thread has moved toward optimizing readers and writers for
fixed-size-list data stored using LIST. Those optimizations provide a good
compatible short-term path, but LIST still models variable-size-list data,
writes redundant repetition levels, and imposes unnecessary writer and
reader complexity and storage overhead.
I propose extending the schema as follows:
```thrift
enum FieldRepetitionType {
REQUIRED = 0,
OPTIONAL = 1,
REPEATED = 2,
VECTOR = 3
}
struct SchemaElement {
// Existing fields...
// Required when repetition_type == VECTOR
optional i32 vector_length;
}
```
VECTOR would mean that a field occurs exactly vector_length times for each
parent value. Unlike REPEATED, the VECTOR dimension would not increase
either the maximum repetition level or the maximum definition
level.
For example, a nullable 768-dimensional float vector could be represented
as:
```text
optional group embedding {
vector float element [768];
}
```
The optional group controls the nullability of the vector as a whole. The
VECTOR child describes 768 dense, non-null float elements.
The main properties of this representation would be:
- The VECTOR dimension produces no repetition levels.
- Elements remain a normal Parquet column, so existing encodings such as
PLAIN, BYTE_STREAM_SPLIT, dictionary encoding, and ALP can operate on the
element type.
- Every present vector contains exactly vector_length contiguous elements.
- A vector must not be split across data pages.
- Nullable vectors use the existing definition-level mechanism. I propose
padded definition-level semantics so that each parent row still occupies
vector_length positions.
- For an initial version, VECTOR elements would be non-null. VECTOR nesting
could be supported, while variable-length repeated descendants would be
excluded.
- Statistics remain element-level; vector-level ordering and min/max
statistics would not be defined.
This is intentionally a breaking format addition. Readers that do not
recognize VECTOR cannot safely interpret the column and must reject it. We
therefore need an explicit compatibility or versioning story.
Compared with an annotated LIST, this representation removes redundant
levels instead of merely allowing optimized readers to ignore them. It also
preserves element-wise encodings and avoids retaining variable-length LIST
semantics as part of the long-term vector representation.
The VECTOR logical type is a related but separate concern. It could
annotate both today’s LIST representation and this native representation,
allowing applications to identify vector semantics independently of how the
values are physically serialized.
The main questions I would like feedback on are:
1. Are the proposed level and null-vector semantics appropriate?
2. What nesting should be supported initially?
3. How should num_values, page indexes, offset indexes, and null counts be
defined?
4. How should unsupported readers detect and reject this representation?
Relevant material:
- Existing discussion:
https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3
- Design document:
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit
- Format prototype: https://github.com/apache/parquet-format/pull/592
- Arrow Go prototype: https://github.com/apache/arrow-go/pull/854
Best,
Rok
