AFAICT, option C has the downside that a optional FIXED_SIZE_LIST would
have a max definition level of 2, not 1. This can have significant
effects on the performance of decoding definition levels.
Also, option C is only a "small amount of work to implement" if you
don't care about optimizing away the unnecessary processing of
repetition levels. If you do, then it's not obvious that it would be
less work to implement than option B.
Regards
Antoine.
Le 01/07/2026 à 20:35, Andrew McCormick via dev a écrit :
Hi Rok,
Just posted a comment to the doc but wanted to add it here, plus add a
little bit of extra info.
Option C looks like by far the strongest option to me. Here's my take:
Option A: not backwards compatible, poor encodings -- little upside.
Option B: clearly the path we'd take if we were designing a new format, but
for parquet as-is it would require a tremendous amount of work, and that
alone makes it untenable unless the gain it gives is way better than other
options.
Option C: Fully backwards compatible, keeps full encodings, small amount of
work to implement on readers and writers. The only downside is that we
still have to store the rep levels on disk (and load them), but due to the
fixed length arrays they compress to almost nothing under the RLE
encodings, so the cost is tiny.
The extra info I wanted to add is that even for files without anything
added to them, you can in fact cheaply detect whether the array in question
is of a fixed size by doing some logic on the compressed rle data. On an
example dataset I benchmarked (100k rows of 4k-float array features) I
measured the decompression time to be 2.3x faster than our baseline reading
path. That verification does of course cost some time, and if we had the
hint from C in the data we could skip it, giving us another 1.5x.
On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> wrote:
---------- Forwarded message ---------
发件人： Rok Mihevc <[email protected]>
Date: 2026年6月29日周一 20:13
Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type
To: <[email protected]>
Cc: <[email protected]>
Hi all,
Based on benchmarks [1] comparing implementation variants, feedback
received via
the design doc [2] and in-person feedback I would like to move forward with
the approach that introduces a new repetition type (Option B in the design
doc and benchmarks) unless there are objections. Either here or on this
week's community call would be a good venue to raise early objections
or provide feedback on how to proceed.
To show what a new repetition type would look like I've opened a
parquet-format PR [3] and a draft Go implementation [4].
[1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef
[2]
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0
[3] https://github.com/apache/parquet-format/pull/592
[4] https://github.com/apache/arrow-go/pull/854
Rok
On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> wrote:
Hi all,
A short update on the progress of this work. State of discussion can be
seen here [1].
I've created a set of naive C++ implementations of the discussed designs;
see here: https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef
Results should be taken with a grain of salt and more of a directional
rather than quantitative information.
Personally I'm leaning towards option B because it is quite expressive
while still providing significant storage and writing performance
improvement.
[1]
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing
[2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef -
benchmarks
[3] https://github.com/rok/arrow/pull/53 - option A
[4] https://github.com/rok/arrow/pull/51 - option B
[5] https://github.com/rok/arrow/pull/52 - option C
Rok
On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> wrote:
Hi all,
Picking this thread back up. I've put together a design doc outlining
three options we've discussed:
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing
* Option A: logical type annotating FIXED_LEN_BYTE_ARRAY.
* Option B: new VECTOR repetition type.
* Option C: logical type annotating a normal LIST, where a recognizing
reader skips rep-level decode and an unknown reader still sees a working
LIST. A future revision would let writers omit rep-levels entirely.
The document evaluates these against the same requirements and compares
them along six axes (backwards compatibility, composability, encoding
flexibility, implementation complexity, on-disk overhead, read
performance). The doc aims to centralize the discussion and help us
pick a
direction.
Comments are open. Most useful pushback would be on the requirements
(especially the "no-fallback breaks adoption" one).
Best,
Rok
On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]>
wrote:
Hello,
The downside with this approach is that the top-level "unit" type is
not
the element type.
For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the
top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that
specialized encodings such as BYTE_STREAM_SPLIT, DELTA_BINARY_PACKED or
ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be
applicable at all (for the latter two).
I wonder if we can find an approach that doesn't emit repetition levels
but still allows using efficient encodings for the element type.
Regards
Antoine.
Le 03/03/2026 à 01:13, Rok Mihevc a écrit :
Hi all,
I'd like to resurrect this thread in light of recent vectors in
Parquet
discussion [1].
There is a (now updated) proposal PR from when the thread was started
that
has a nice discussion [2].
TLDR of the current proposal:
- FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf
with
FixedSizeListType { type, num_values }.
- type must be fixed-width and non-array (INT32, INT64, FLOAT,
DOUBLE,
FIXED_LEN_BYTE_ARRAY); num_values > 0.
- type_length must match num_values encoded with PLAIN representation
of
type.
- If the field is optional, the whole list value may be null;
elements
are
always non-null.
- Intentionally not a `LIST` encoding (no def/rep levels).
- Outer page/column encoding behavior is unchanged (any encoding
valid
for
`FIXED_LEN_BYTE_ARRAY` remains valid).
I also added explicit validity requirements: writers must not emit
violating metadata, and readers must treat violating metadata as
invalid.
Rok
[1] https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3
[2] https://github.com/apache/parquet-format/pull/241
On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> wrote:
I would love to see this!
It is an important optimization for vectors, which become more and
more
important and ubiquitous for grounding of LLMs.
Note however that the logical type route has one drawback: A logical
type
may not change the physical representation of values! Thus, if we
make
FIXED_SIZE_LIST just a logical type, we would still need to write
R-Levels,
as even clients not knowing this logical type need to be able to
decode the
column. We could avoid reading the R-Levels and just assume that
each
list
has the fixed size, so the read path would be optimized but the
write
path
wouldn't.
If we want to avoid writing R-Levels altogether, a logical type
doesn't cut
it. It needs to be something different. E.g., in the schema, we
could
store
an optional `count` for repeated fields. Whenever this count is
present, we
would not write R-Levels for this field (or more precisely, this
field
would not take part in the R-Level computation, as if it wasn't a
repeated
field). This of course is a more intrusive change, as legacy clients
couldn't read such columns anymore.
I don't know which of the two alternatives is better. I agree with
Gang
that we should probably discuss this in a PR.
Cheers,
Jan
Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu <[email protected]
:
Hi Rok,
Happy to see you here :)
According to my past experience, it would be more helpful to open
a PR against the parquet-format repository and post it here.
Best,
Gang
On Wed, May 15, 2024 at 7:25 PM Rok Mihevc <[email protected]>
wrote:
Hi all,
Arrow recently introduced FixedShapeTensor and VariableShapeTensor
canonical extension types [1] that use FixedSizeList and
StructArray(List,
FixedSizeList) as storage respectfully. These are targeted at
machine
learning and scientific applications that deal with large datasets
and
would benefit from using Parquet as on disk storage.
However currently FixedSizeList is stored as List in Parquet which
adds
significant conversion overhead when reading and writing [2]. It
would
therefore be beneficial to introduce a FIXED_SIZE_LIST logical
type.
I would like to open a discussion on potentially adding
FIXED_SIZE_LIST
type and prepare a proposal if discussion supports it.
Best,
Rok
[1]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list
[2] https://github.com/apache/arrow/issues/34510
