We have implemented Option C and the work is a couple of hundred lines of code. I estimate the Option B will have about the same complexity. The benefit of Option C is that we can ship it asap and we have no backward or forward compatibility to think about.
On Mon, Jul 6, 2026 at 2:24 PM Antoine Pitrou <[email protected]> wrote: > > AFAICT, option C has the downside that a optional FIXED_SIZE_LIST would > have a max definition level of 2, not 1. This can have significant > effects on the performance of decoding definition levels. > > Also, option C is only a "small amount of work to implement" if you > don't care about optimizing away the unnecessary processing of > repetition levels. If you do, then it's not obvious that it would be > less work to implement than option B. > > Regards > > Antoine. > > > Le 01/07/2026 à 20:35, Andrew McCormick via dev a écrit : > > Hi Rok, > > > > Just posted a comment to the doc but wanted to add it here, plus add a > > little bit of extra info. > > > > Option C looks like by far the strongest option to me. Here's my take: > > > > Option A: not backwards compatible, poor encodings -- little upside. > > Option B: clearly the path we'd take if we were designing a new format, > but > > for parquet as-is it would require a tremendous amount of work, and that > > alone makes it untenable unless the gain it gives is way better than > other > > options. > > Option C: Fully backwards compatible, keeps full encodings, small amount > of > > work to implement on readers and writers. The only downside is that we > > still have to store the rep levels on disk (and load them), but due to > the > > fixed length arrays they compress to almost nothing under the RLE > > encodings, so the cost is tiny. > > > > The extra info I wanted to add is that even for files without anything > > added to them, you can in fact cheaply detect whether the array in > question > > is of a fixed size by doing some logic on the compressed rle data. On an > > example dataset I benchmarked (100k rows of 4k-float array features) I > > measured the decompression time to be 2.3x faster than our baseline > reading > > path. That verification does of course cost some time, and if we had the > > hint from C in the data we could skip it, giving us another 1.5x. > > > > On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> wrote: > > > > > > > >> > >> ---------- Forwarded message --------- > >> 发件人： Rok Mihevc <[email protected]> > >> Date: 2026年6月29日周一 20:13 > >> Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type > >> To: <[email protected]> > >> Cc: <[email protected]> > >> > >> > >> Hi all, > >> > >> Based on benchmarks [1] comparing implementation variants, feedback > >> received via > >> the design doc [2] and in-person feedback I would like to move forward > with > >> the approach that introduces a new repetition type (Option B in the > design > >> doc and benchmarks) unless there are objections. Either here or on this > >> week's community call would be a good venue to raise early objections > >> or provide feedback on how to proceed. > >> > >> To show what a new repetition type would look like I've opened a > >> parquet-format PR [3] and a draft Go implementation [4]. > >> > >> [1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > >> [2] > >> > >> > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 > >> [3] https://github.com/apache/parquet-format/pull/592 > >> [4] https://github.com/apache/arrow-go/pull/854 > >> > >> Rok > >> > >> On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> > wrote: > >> > >>> Hi all, > >>> > >>> A short update on the progress of this work. State of discussion can be > >>> seen here [1]. > >>> I've created a set of naive C++ implementations of the discussed > designs; > >>> see here: https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > >>> Results should be taken with a grain of salt and more of a directional > >>> rather than quantitative information. > >>> > >>> Personally I'm leaning towards option B because it is quite expressive > >>> while still providing significant storage and writing performance > >>> improvement. > >>> > >>> [1] > >>> > >> > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > >>> [2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef - > >>> benchmarks > >>> [3] https://github.com/rok/arrow/pull/53 - option A > >>> [4] https://github.com/rok/arrow/pull/51 - option B > >>> [5] https://github.com/rok/arrow/pull/52 - option C > >>> > >>> Rok > >>> > >>> On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> > wrote: > >>> > >>>> Hi all, > >>>> > >>>> Picking this thread back up. I've put together a design doc outlining > >>>> three options we've discussed: > >>>> > >>>> > >> > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > >>>> > >>>> * Option A: logical type annotating FIXED_LEN_BYTE_ARRAY. > >>>> * Option B: new VECTOR repetition type. > >>>> * Option C: logical type annotating a normal LIST, where a recognizing > >>>> reader skips rep-level decode and an unknown reader still sees a > working > >>>> LIST. A future revision would let writers omit rep-levels entirely. > >>>> > >>>> The document evaluates these against the same requirements and > compares > >>>> them along six axes (backwards compatibility, composability, encoding > >>>> flexibility, implementation complexity, on-disk overhead, read > >>>> performance). The doc aims to centralize the discussion and help us > >> pick a > >>>> direction. > >>>> Comments are open. Most useful pushback would be on the requirements > >>>> (especially the "no-fallback breaks adoption" one). > >>>> > >>>> Best, > >>>> Rok > >>>> > >>>> On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]> > >> wrote: > >>>> > >>>>> > >>>>> Hello, > >>>>> > >>>>> The downside with this approach is that the top-level "unit" type is > >> not > >>>>> the element type. > >>>>> > >>>>> For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the > >>>>> top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that > >>>>> specialized encodings such as BYTE_STREAM_SPLIT, DELTA_BINARY_PACKED > or > >>>>> ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be > >>>>> applicable at all (for the latter two). > >>>>> > >>>>> I wonder if we can find an approach that doesn't emit repetition > levels > >>>>> but still allows using efficient encodings for the element type. > >>>>> > >>>>> Regards > >>>>> > >>>>> Antoine. > >>>>> > >>>>> > >>>>> Le 03/03/2026 à 01:13, Rok Mihevc a écrit : > >>>>>> Hi all, > >>>>>> > >>>>>> I'd like to resurrect this thread in light of recent vectors in > >> Parquet > >>>>>> discussion [1]. > >>>>>> There is a (now updated) proposal PR from when the thread was > started > >>>>> that > >>>>>> has a nice discussion [2]. > >>>>>> > >>>>>> TLDR of the current proposal: > >>>>>> - FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf > >> with > >>>>>> FixedSizeListType { type, num_values }. > >>>>>> - type must be fixed-width and non-array (INT32, INT64, FLOAT, > >> DOUBLE, > >>>>>> FIXED_LEN_BYTE_ARRAY); num_values > 0. > >>>>>> - type_length must match num_values encoded with PLAIN > representation > >>>>> of > >>>>>> type. > >>>>>> - If the field is optional, the whole list value may be null; > >> elements > >>>>> are > >>>>>> always non-null. > >>>>>> - Intentionally not a `LIST` encoding (no def/rep levels). > >>>>>> - Outer page/column encoding behavior is unchanged (any encoding > >> valid > >>>>> for > >>>>>> `FIXED_LEN_BYTE_ARRAY` remains valid). > >>>>>> > >>>>>> I also added explicit validity requirements: writers must not emit > >>>>>> violating metadata, and readers must treat violating metadata as > >>>>> invalid. > >>>>>> > >>>>>> > >>>>>> Rok > >>>>>> > >>>>>> [1] > https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3 > >>>>>> [2] https://github.com/apache/parquet-format/pull/241 > >>>>>> > >>>>>> On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> > wrote: > >>>>>> > >>>>>>> I would love to see this! > >>>>>>> > >>>>>>> It is an important optimization for vectors, which become more and > >>>>> more > >>>>>>> important and ubiquitous for grounding of LLMs. > >>>>>>> > >>>>>>> Note however that the logical type route has one drawback: A > logical > >>>>> type > >>>>>>> may not change the physical representation of values! Thus, if we > >> make > >>>>>>> FIXED_SIZE_LIST just a logical type, we would still need to write > >>>>> R-Levels, > >>>>>>> as even clients not knowing this logical type need to be able to > >>>>> decode the > >>>>>>> column. We could avoid reading the R-Levels and just assume that > >> each > >>>>> list > >>>>>>> has the fixed size, so the read path would be optimized but the > >> write > >>>>> path > >>>>>>> wouldn't. > >>>>>>> > >>>>>>> If we want to avoid writing R-Levels altogether, a logical type > >>>>> doesn't cut > >>>>>>> it. It needs to be something different. E.g., in the schema, we > >> could > >>>>> store > >>>>>>> an optional `count` for repeated fields. Whenever this count is > >>>>> present, we > >>>>>>> would not write R-Levels for this field (or more precisely, this > >> field > >>>>>>> would not take part in the R-Level computation, as if it wasn't a > >>>>> repeated > >>>>>>> field). This of course is a more intrusive change, as legacy > clients > >>>>>>> couldn't read such columns anymore. > >>>>>>> > >>>>>>> I don't know which of the two alternatives is better. I agree with > >>>>> Gang > >>>>>>> that we should probably discuss this in a PR. > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Jan > >>>>>>> > >>>>>>> > >>>>>>> Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu < > [email protected] > >>> : > >>>>>>> > >>>>>>>> Hi Rok, > >>>>>>>> > >>>>>>>> Happy to see you here :) > >>>>>>>> > >>>>>>>> According to my past experience, it would be more helpful to open > >>>>>>>> a PR against the parquet-format repository and post it here. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Gang > >>>>>>>> > >>>>>>>> On Wed, May 15, 2024 at 7:25 PM Rok Mihevc <[email protected]> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi all, > >>>>>>>>> > >>>>>>>>> Arrow recently introduced FixedShapeTensor and > VariableShapeTensor > >>>>>>>>> canonical extension types [1] that use FixedSizeList and > >>>>>>>> StructArray(List, > >>>>>>>>> FixedSizeList) as storage respectfully. These are targeted at > >>>>> machine > >>>>>>>>> learning and scientific applications that deal with large > datasets > >>>>> and > >>>>>>>>> would benefit from using Parquet as on disk storage. > >>>>>>>>> > >>>>>>>>> However currently FixedSizeList is stored as List in Parquet > which > >>>>> adds > >>>>>>>>> significant conversion overhead when reading and writing [2]. It > >>>>> would > >>>>>>>>> therefore be beneficial to introduce a FIXED_SIZE_LIST logical > >> type. > >>>>>>>>> > >>>>>>>>> I would like to open a discussion on potentially adding > >>>>> FIXED_SIZE_LIST > >>>>>>>>> type and prepare a proposal if discussion supports it. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Rok > >>>>>>>>> > >>>>>>>>> [1] > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >> > https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list > >>>>>>>>> [2] https://github.com/apache/arrow/issues/34510 > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >> > > > > >
