Very neat! Very very neat :D I think this result deserves some reflection beyond "nice interim optimization," though. I'm genuinely unsure what remains of the performance case for a new logical type, as opposed to the narrower questions of null-stride semantics and Arrow round-trip fidelity (which are worth discussing on their own merits).
More generally, it makes me think the highest-leverage format-evolution work right now is at the encoding and reader layer - ALP and friends - rather than the type layer. Readers doing the work the encoding already permits, as Hardwood just did, benefits every existing file; a new logical type benefits only files written after the multi-year adoption lag. It will be interesting to see whether the def-level gate alone (the O(1) "no nulls in this page" check) gets picked up by other readers — that part seems universally applicable, well beyond fixed lists. Thanks for sharing this, Will On Wed, 22 Jul 2026 at 15:03, Gunnar Morling <[email protected]> wrote: > Hey all, > > As briefly mentioned before, we've implemented a fixed-length list > fast path in Hardwood; it detects effectively fixed-length lists by > scanning the encoded def/rep level streams and bypasses the standard > Dremel reconstruction in this case. For larger lists, e.g. 768-element > vector embeddings, this brings read times to the level of a flat > column with the same data. > > I just published a blog post about it, perhaps it's interesting to folks > here: > > > https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/ > > I think this can be a nice interim read-side optimization until a > native FIXED_SIZE_LIST type lands in the spec. Would love to hear from > others too if they apply similar techniques. > > All the best, > > --Gunnar > > On Sat, 11 Jul 2026 at 02:06, Rok Mihevc <[email protected]> wrote: > > > > Hi all, > > > > Thanks everyone for the discussion so far! Let me summarize where I > believe > > we are and propose a variant of Option B. > > > > The three options represent vectors differently at the schema and > physical > > levels: > > > > Option A > > - Proposal: represent vectors using FLBA > > - Pros: simple and efficient, close to representations used by > existing > > engines > > - Cons: limited composability, no element nullability, and no natural > > support for element-wise encodings > > Option B > > - Proposal: add a VECTOR repetition type to allow the VECTOR > dimension > > to be omitted from repetition levels > > - Pros: avoids storing and processing redundant repetition levels, > > performs best on writes in current prototypes > > - Cons: introduces a new repetition type, with unclear compatibility > > and implementation impact. (can we estimate this somehow??) > > Option C > > - Proposal: annotate the existing LIST representation with a logical > > type containing the vector length > > - Pros: allows optimizing readers to avoid reading redundant > repetition > > levels, Andrew's prototype shows read performance can match Option B, > > strongest compatibility, and implementation impact, optimizations can > > follow at pace they choose > > - Cons: redundant repetition levels are written > > > > > > > On Mon, Jul 6, 2026 at 5:03 PM Alkis Evlogimenos < > > [email protected]> wrote: > > > > > > 1. The logical type makes the implementation that assumes all arrays > are > > fixed length, on par in performance with Option B > > > 2. Without the logical type (and with a little bit extra complexity) a > > smart enough *reader* can walk the def/rep levels before decoding, infer > > (1) - the writer wrote fixed len arrays - and call the implementation in > > (1). Compared to (1) this is 1.5x slower. > > > > Then nothing is stopping implementations from doing this today? Gunnar > > already merged this. We should document it as a useful implementation > > optimization. > > > > > > I have been thinking about whether we could obtain Option B-like levels > > without introducing a new repetition type. One possibility would be to > add > > an optional i32 length field to SchemaElement, allowing us to represent > > types such as: > > > > optional float embedding[3]; > > > > Here, optional controls the nullability of the entire vector, while > length > > = 3 means that every present vector contains exactly three floats. The > > fixed dimension would not contribute > > a repetition level. The presence of length would itself carry the > > fixed-vector semantics, so a separate VECTOR logical annotation may not > be > > necessary. > > > > Given the following three vectors: > > > > [1.0, 2.0, 3.0] > > null > > [4.0, 5.0, 6.0] > > > > the physical representation would be: > > > > repetition levels: omitted > > definition levels: [1, 0, 1] > > physical values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] > > > > Reader would use SchemaElement.length=3 to interpret this. VECTOR-unaware > > readers must therefore reject the vector columns. One way to provide that > > gate would be to require VECTOR-specific page encodings, such as > > VECTOR_PLAIN and VECTOR_BYTE_STREAM_SPLIT, on pages with such a column. A > > VECTOR-unaware reader would reject the unsupported encoding before > > attempting to decode the page. > > > > The other possible gating mechanism is, of course, versioning. :) > > > > The primary purpose of a VECTOR encoding here would be to gate out > > VECTOR-unaware readers cleanly. There may also be another motivation: > > vector similarity workloads may benefit from storing values by > > dimension rather than by vector. > > > > For M vectors of length N: > > - By vector: v0[0] ... v0[N], v1[0] ... v1[N], ... > > - By dimension: v0[0] ... vM[0], v0[1] ... vM[1], ... > > > > This layout is discussed in the PDX paper [1] and would naturally be > > represented as a page encoding in parquet. > > > > Would an augmented Option B along these lines be worth exploring? > > > > Rok > > > > [1] https://arxiv.org/pdf/2503.04422 > > > > On Fri, Jul 10, 2026 at 7:02 PM Andrew McCormick via dev < > > [email protected]> wrote: > > > > > Hi Antoine, > > > > > > I wrote some more code to answer your question, details here: > > > https://termbin.com/0zzc8. > > > > > > I added a nullability knob to the benchmark that writes an optional > outer > > > array with zero nulls actually present (your "empty lists need \ > > > a def level" case, but at 100% presence) and verified on disk that this > > > really is maxDef=2 (asserted in a test). The results are: > > > arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) delta > > > B/VECTOR 1401 +/- 6 1403 +/- 12 +0.2% > > > (same required file; no nullable form) > > > C-hint 1436 +/- 8 1428 +/- 8 -0.6% > > > (annotation-aware skip; FLAT) > > > C-naive 2606 +/- 11 2603 +/- 10 -0.1% > > > (full Dremel decode) > > > LIST 2613 +/- 11 2617 +/- 13 +0.1% > > > > > > Note that the option B number above isn't actually possible with the > > > optional array, since Rok's impl only kicks in on required arrays, but > I > > > included it here for reference to show the optional array doesn't slow > > > option C down. > > > > > > > > > On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> > wrote: > > > > > > > > > > > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit : > > > > > I built a prototype of hint-supported reads for option C on top of > > > Rok's > > > > > work. Here's the results I see: > > > > > > > > > > arm ns/row (mean +/- sd) note > > > > > A/FLBA 2730 +/- 13 no levels on disk > (FLBA->float > > > > > reinterpret adds a bit) > > > > > B/VECTOR 2337 +/- 10 no levels, not forward > compatible > > > > > C-hint 2356 +/- 5 skip-levels reader on a plain > > > > > LIST; fully backward-compatible > > > > > C-dremel 3830 +/- 22 annotation ignored, full > Dremel > > > > > (aka what Rok measured) > > > > > > > > > > So basically when you use the hint C is within noise of B (<1%). > Full > > > > > details and code here: https://termbin.com/kj2x > > > > > (gist isn't availble on my db github). > > > > > > > > One remaining question is what happens for definition levels (not > > > > repetition) in option C. Empty lists need a specific definition > level to > > > > encode, therefore option C makes the max def level larger than > option B. > > > > > > > > An optional column with option C might therefore take more time > decoding > > > > than option B (especially if crossing the threshold from 1-bit to > 2-bit > > > > levels). > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling < > > > > [email protected]> > > > > > wrote: > > > > > > > > > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]> > > > wrote: > > > > >>> > > > > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit : > > > > >>>>>> 2. Without the logical type (and with a little bit extra > > > complexity) > > > > >> a > > > > >>>>>> smart enough *reader* can walk the def/rep levels before > decoding, > > > > >> infer > > > > >>>> > > > > >>>>> At the cost of higher implementation complexity and maintenance > > > cost. > > > > >>>>> Does any mainstream open source implementation of Parquet do > this? > > > > >>>> > > > > >>>> Triggered by the conversation on the call last week, I > implemented > > > > >>>> pretty much this in Hardwood [1]. > > > > >>> > > > > >>> Great, thank you. `FixedSizeListDetector.java` is highly > non-trivial > > > > and > > > > >>> definitely has a maintenance cost. Though part of it seems about > not > > > > >>> having a RLE parser abstraction available. > > > > >> > > > > >> Yes, I think we all agree that a dedicated type will make > maintainers' > > > > >> lives much easier and is the right solution eventually. But until > that > > > > >> has landed, I think there's some juice worth the squeeze here. > > > > >> > > > > >>> > > > > >>> Regards > > > > >>> > > > > >>> Antoine. > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > > > >
