Here's why Option C is the superior option: 1. The logical type makes the implementation that assumes all arrays are fixed length, on par in performance with Option B 2. Without the logical type (and with a little bit extra complexity) a smart enough *reader* can walk the def/rep levels before decoding, infer (1) - the writer wrote fixed len arrays - and call the implementation in (1). Compared to (1) this is 1.5x slower. 3. Without the logical type a naive reader - that is any reader of today - can decode the fixed len arrays. Compared to (1) this is 5x slower.
The above are prototyped and benchmarked on Databricks Photon engine (C++). This is why I claim that Option C can roll out *today*. We ratify the spec change asap. Readers/writers can move on their own pace to unlock each of the performance gates at a complexity/performance tradeoff of their choosing. Some readers will ignore the logical type completely and opt to not do any optimizations, others will do only (1), others will do both (1) and (2). On Mon, Jul 6, 2026 at 4:38 PM Rok Mihevc <[email protected]> wrote: > So if the implementation effort difference is marginal, compatibility > issues can be handled with versioning, and vector columns are arguably a > niche type (limiting the blast radius), why would we consider a less > optimal approach? Are there other reasons we haven't discussed yet? > > In either case we need to introduce a logical type, bringing us to about the > same time frame for changing the spec. > > Rok > > On Mon, Jul 6, 2026 at 3:08 PM Alkis Evlogimenos via dev < > [email protected]> wrote: > >> We have implemented Option C and the work is a couple of hundred lines of >> code. I estimate the Option B will have about the same complexity. The >> benefit of Option C is that we can ship it asap and we have no backward or >> forward compatibility to think about. >> >> On Mon, Jul 6, 2026 at 2:24 PM Antoine Pitrou <[email protected]> wrote: >> >> > >> > AFAICT, option C has the downside that a optional FIXED_SIZE_LIST would >> > have a max definition level of 2, not 1. This can have significant >> > effects on the performance of decoding definition levels. >> > >> > Also, option C is only a "small amount of work to implement" if you >> > don't care about optimizing away the unnecessary processing of >> > repetition levels. If you do, then it's not obvious that it would be >> > less work to implement than option B. >> > >> > Regards >> > >> > Antoine. >> > >> > >> > Le 01/07/2026 à 20:35, Andrew McCormick via dev a écrit : >> > > Hi Rok, >> > > >> > > Just posted a comment to the doc but wanted to add it here, plus add a >> > > little bit of extra info. >> > > >> > > Option C looks like by far the strongest option to me. Here's my take: >> > > >> > > Option A: not backwards compatible, poor encodings -- little upside. >> > > Option B: clearly the path we'd take if we were designing a new >> format, >> > but >> > > for parquet as-is it would require a tremendous amount of work, and >> that >> > > alone makes it untenable unless the gain it gives is way better than >> > other >> > > options. >> > > Option C: Fully backwards compatible, keeps full encodings, small >> amount >> > of >> > > work to implement on readers and writers. The only downside is that we >> > > still have to store the rep levels on disk (and load them), but due to >> > the >> > > fixed length arrays they compress to almost nothing under the RLE >> > > encodings, so the cost is tiny. >> > > >> > > The extra info I wanted to add is that even for files without anything >> > > added to them, you can in fact cheaply detect whether the array in >> > question >> > > is of a fixed size by doing some logic on the compressed rle data. On >> an >> > > example dataset I benchmarked (100k rows of 4k-float array features) I >> > > measured the decompression time to be 2.3x faster than our baseline >> > reading >> > > path. That verification does of course cost some time, and if we had >> the >> > > hint from C in the data we could skip it, giving us another 1.5x. >> > > >> > > On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> >> wrote: >> > > >> > > >> > > >> > >> >> > >> ---------- Forwarded message --------- >> > >> 发件人： Rok Mihevc <[email protected]> >> > >> Date: 2026年6月29日周一 20:13 >> > >> Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type >> > >> To: <[email protected]> >> > >> Cc: <[email protected]> >> > >> >> > >> >> > >> Hi all, >> > >> >> > >> Based on benchmarks [1] comparing implementation variants, feedback >> > >> received via >> > >> the design doc [2] and in-person feedback I would like to move >> forward >> > with >> > >> the approach that introduces a new repetition type (Option B in the >> > design >> > >> doc and benchmarks) unless there are objections. Either here or on >> this >> > >> week's community call would be a good venue to raise early objections >> > >> or provide feedback on how to proceed. >> > >> >> > >> To show what a new repetition type would look like I've opened a >> > >> parquet-format PR [3] and a draft Go implementation [4]. >> > >> >> > >> [1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef >> > >> [2] >> > >> >> > >> >> > >> https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 >> > >> [3] https://github.com/apache/parquet-format/pull/592 >> > >> [4] https://github.com/apache/arrow-go/pull/854 >> > >> >> > >> Rok >> > >> >> > >> On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> >> > wrote: >> > >> >> > >>> Hi all, >> > >>> >> > >>> A short update on the progress of this work. State of discussion >> can be >> > >>> seen here [1]. >> > >>> I've created a set of naive C++ implementations of the discussed >> > designs; >> > >>> see here: >> https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef >> > >>> Results should be taken with a grain of salt and more of a >> directional >> > >>> rather than quantitative information. >> > >>> >> > >>> Personally I'm leaning towards option B because it is quite >> expressive >> > >>> while still providing significant storage and writing performance >> > >>> improvement. >> > >>> >> > >>> [1] >> > >>> >> > >> >> > >> https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing >> > >>> [2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef - >> > >>> benchmarks >> > >>> [3] https://github.com/rok/arrow/pull/53 - option A >> > >>> [4] https://github.com/rok/arrow/pull/51 - option B >> > >>> [5] https://github.com/rok/arrow/pull/52 - option C >> > >>> >> > >>> Rok >> > >>> >> > >>> On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> >> > wrote: >> > >>> >> > >>>> Hi all, >> > >>>> >> > >>>> Picking this thread back up. I've put together a design doc >> outlining >> > >>>> three options we've discussed: >> > >>>> >> > >>>> >> > >> >> > >> https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing >> > >>>> >> > >>>> * Option A: logical type annotating FIXED_LEN_BYTE_ARRAY. >> > >>>> * Option B: new VECTOR repetition type. >> > >>>> * Option C: logical type annotating a normal LIST, where a >> recognizing >> > >>>> reader skips rep-level decode and an unknown reader still sees a >> > working >> > >>>> LIST. A future revision would let writers omit rep-levels entirely. >> > >>>> >> > >>>> The document evaluates these against the same requirements and >> > compares >> > >>>> them along six axes (backwards compatibility, composability, >> encoding >> > >>>> flexibility, implementation complexity, on-disk overhead, read >> > >>>> performance). The doc aims to centralize the discussion and help us >> > >> pick a >> > >>>> direction. >> > >>>> Comments are open. Most useful pushback would be on the >> requirements >> > >>>> (especially the "no-fallback breaks adoption" one). >> > >>>> >> > >>>> Best, >> > >>>> Rok >> > >>>> >> > >>>> On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]> >> > >> wrote: >> > >>>> >> > >>>>> >> > >>>>> Hello, >> > >>>>> >> > >>>>> The downside with this approach is that the top-level "unit" type >> is >> > >> not >> > >>>>> the element type. >> > >>>>> >> > >>>>> For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the >> > >>>>> top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that >> > >>>>> specialized encodings such as BYTE_STREAM_SPLIT, >> DELTA_BINARY_PACKED >> > or >> > >>>>> ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be >> > >>>>> applicable at all (for the latter two). >> > >>>>> >> > >>>>> I wonder if we can find an approach that doesn't emit repetition >> > levels >> > >>>>> but still allows using efficient encodings for the element type. >> > >>>>> >> > >>>>> Regards >> > >>>>> >> > >>>>> Antoine. >> > >>>>> >> > >>>>> >> > >>>>> Le 03/03/2026 à 01:13, Rok Mihevc a écrit : >> > >>>>>> Hi all, >> > >>>>>> >> > >>>>>> I'd like to resurrect this thread in light of recent vectors in >> > >> Parquet >> > >>>>>> discussion [1]. >> > >>>>>> There is a (now updated) proposal PR from when the thread was >> > started >> > >>>>> that >> > >>>>>> has a nice discussion [2]. >> > >>>>>> >> > >>>>>> TLDR of the current proposal: >> > >>>>>> - FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf >> > >> with >> > >>>>>> FixedSizeListType { type, num_values }. >> > >>>>>> - type must be fixed-width and non-array (INT32, INT64, FLOAT, >> > >> DOUBLE, >> > >>>>>> FIXED_LEN_BYTE_ARRAY); num_values > 0. >> > >>>>>> - type_length must match num_values encoded with PLAIN >> > representation >> > >>>>> of >> > >>>>>> type. >> > >>>>>> - If the field is optional, the whole list value may be null; >> > >> elements >> > >>>>> are >> > >>>>>> always non-null. >> > >>>>>> - Intentionally not a `LIST` encoding (no def/rep levels). >> > >>>>>> - Outer page/column encoding behavior is unchanged (any encoding >> > >> valid >> > >>>>> for >> > >>>>>> `FIXED_LEN_BYTE_ARRAY` remains valid). >> > >>>>>> >> > >>>>>> I also added explicit validity requirements: writers must not >> emit >> > >>>>>> violating metadata, and readers must treat violating metadata as >> > >>>>> invalid. >> > >>>>>> >> > >>>>>> >> > >>>>>> Rok >> > >>>>>> >> > >>>>>> [1] >> > https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3 >> > >>>>>> [2] https://github.com/apache/parquet-format/pull/241 >> > >>>>>> >> > >>>>>> On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> >> > wrote: >> > >>>>>> >> > >>>>>>> I would love to see this! >> > >>>>>>> >> > >>>>>>> It is an important optimization for vectors, which become more >> and >> > >>>>> more >> > >>>>>>> important and ubiquitous for grounding of LLMs. >> > >>>>>>> >> > >>>>>>> Note however that the logical type route has one drawback: A >> > logical >> > >>>>> type >> > >>>>>>> may not change the physical representation of values! Thus, if >> we >> > >> make >> > >>>>>>> FIXED_SIZE_LIST just a logical type, we would still need to >> write >> > >>>>> R-Levels, >> > >>>>>>> as even clients not knowing this logical type need to be able to >> > >>>>> decode the >> > >>>>>>> column. We could avoid reading the R-Levels and just assume that >> > >> each >> > >>>>> list >> > >>>>>>> has the fixed size, so the read path would be optimized but the >> > >> write >> > >>>>> path >> > >>>>>>> wouldn't. >> > >>>>>>> >> > >>>>>>> If we want to avoid writing R-Levels altogether, a logical type >> > >>>>> doesn't cut >> > >>>>>>> it. It needs to be something different. E.g., in the schema, we >> > >> could >> > >>>>> store >> > >>>>>>> an optional `count` for repeated fields. Whenever this count is >> > >>>>> present, we >> > >>>>>>> would not write R-Levels for this field (or more precisely, this >> > >> field >> > >>>>>>> would not take part in the R-Level computation, as if it wasn't >> a >> > >>>>> repeated >> > >>>>>>> field). This of course is a more intrusive change, as legacy >> > clients >> > >>>>>>> couldn't read such columns anymore. >> > >>>>>>> >> > >>>>>>> I don't know which of the two alternatives is better. I agree >> with >> > >>>>> Gang >> > >>>>>>> that we should probably discuss this in a PR. >> > >>>>>>> >> > >>>>>>> Cheers, >> > >>>>>>> Jan >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu < >> > [email protected] >> > >>> : >> > >>>>>>> >> > >>>>>>>> Hi Rok, >> > >>>>>>>> >> > >>>>>>>> Happy to see you here :) >> > >>>>>>>> >> > >>>>>>>> According to my past experience, it would be more helpful to >> open >> > >>>>>>>> a PR against the parquet-format repository and post it here. >> > >>>>>>>> >> > >>>>>>>> Best, >> > >>>>>>>> Gang >> > >>>>>>>> >> > >>>>>>>> On Wed, May 15, 2024 at 7:25 PM Rok Mihevc < >> [email protected]> >> > >>>>> wrote: >> > >>>>>>>> >> > >>>>>>>>> Hi all, >> > >>>>>>>>> >> > >>>>>>>>> Arrow recently introduced FixedShapeTensor and >> > VariableShapeTensor >> > >>>>>>>>> canonical extension types [1] that use FixedSizeList and >> > >>>>>>>> StructArray(List, >> > >>>>>>>>> FixedSizeList) as storage respectfully. These are targeted at >> > >>>>> machine >> > >>>>>>>>> learning and scientific applications that deal with large >> > datasets >> > >>>>> and >> > >>>>>>>>> would benefit from using Parquet as on disk storage. >> > >>>>>>>>> >> > >>>>>>>>> However currently FixedSizeList is stored as List in Parquet >> > which >> > >>>>> adds >> > >>>>>>>>> significant conversion overhead when reading and writing [2]. >> It >> > >>>>> would >> > >>>>>>>>> therefore be beneficial to introduce a FIXED_SIZE_LIST logical >> > >> type. >> > >>>>>>>>> >> > >>>>>>>>> I would like to open a discussion on potentially adding >> > >>>>> FIXED_SIZE_LIST >> > >>>>>>>>> type and prepare a proposal if discussion supports it. >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> Best, >> > >>>>>>>>> Rok >> > >>>>>>>>> >> > >>>>>>>>> [1] >> > >>>>>>>>> >> > >>>>>>>> >> > >>>>>>> >> > >>>>> >> > >> >> > >> https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list >> > >>>>>>>>> [2] https://github.com/apache/arrow/issues/34510 >> > >>>>>>>>> >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >> >> > > >> > >> > >> > >> >
