Hi all, We started this thread by discussing a new physical representation for fixed-size-lists (FSLs), but the recent discussion has shifted toward optimizing readers and writers for fixed-size data already stored as LIST. The prototypes and benchmarks have helped separate three concerns:
1. Short-term implementation: Readers and writers should optimize the existing LIST representation when its values are known or detected to have a fixed size. 2. Logical semantics: We should define a VECTOR logical type that can annotate both the existing LIST representation and a future native fixed-size representation. 3. Long-term physical representation: We should define a native serialization that avoids LIST’s limitations, even if that requires a breaking format change. The logical and physical proposals answer different questions: the logical type describes what the data means, while the physical representation describes how it is serialized. They also have very different compatibility and implementation implications. I therefore suggest discussing them separately. We can continue the logical-type discussion in this thread, including its use with LIST, while discussing the new physical representation in a dedicated thread. This avoids either decision blocking the other: the larger compatibility discussion around a native representation should not delay the logical type, but adopting a logical type or optimizing LIST should not cause work on the long-term physical representation to stall. I will open a separate thread proposing Option B - the new VECTOR repetition type - as that physical representation. Best, Rok On Mon, Jul 27, 2026 at 1:43 PM Gunnar Morling < [email protected]> wrote: > Hey Will, Alkis, all, > > Thanks for the nice feedback! Great to hear that Photon is doing > something similar; would be nice to see other engines adopt this too. > > Beyond this work, I believe there's still value in a having dedicated > logical type for fixed-length vectors. The on-disk format would be be > more efficient (no repetition levels needed at all), decoding could > also be better for those lists lengths currently requiring that scalar > fallback logic. But yes, the urgency is a bit lower than originally > anticipated perhaps. > > Best, > > --Gunnar > > > On Mon, 27 Jul 2026 at 11:28, Alkis Evlogimenos via dev > <[email protected]> wrote: > > > > Thank you Gunnar for pushing this in hardwood and blogging about it. This > > verifies the same work Andrew did internally on Photon. > > > > With the above in mind the fixed size list discussion shifts from > physical > > type changes to parquet to parser quality of implementation. > > > > The logical type discussion is still open: do we need a vector type with > > more restrictive definition of what vector is? > > > > On Sat, Jul 11, 2026 at 3:06 AM Rok Mihevc <[email protected]> wrote: > > > > > Hi all, > > > > > > Thanks everyone for the discussion so far! Let me summarize where I > believe > > > we are and propose a variant of Option B. > > > > > > The three options represent vectors differently at the schema and > physical > > > levels: > > > > > > Option A > > > - Proposal: represent vectors using FLBA > > > - Pros: simple and efficient, close to representations used by > existing > > > engines > > > - Cons: limited composability, no element nullability, and no > natural > > > support for element-wise encodings > > > Option B > > > - Proposal: add a VECTOR repetition type to allow the VECTOR > dimension > > > to be omitted from repetition levels > > > - Pros: avoids storing and processing redundant repetition levels, > > > performs best on writes in current prototypes > > > - Cons: introduces a new repetition type, with unclear > compatibility > > > and implementation impact. (can we estimate this somehow??) > > > Option C > > > - Proposal: annotate the existing LIST representation with a > logical > > > type containing the vector length > > > - Pros: allows optimizing readers to avoid reading redundant > repetition > > > levels, Andrew's prototype shows read performance can match Option B, > > > strongest compatibility, and implementation impact, optimizations can > > > follow at pace they choose > > > - Cons: redundant repetition levels are written > > > > > > > > > > On Mon, Jul 6, 2026 at 5:03 PM Alkis Evlogimenos < > > > [email protected]> wrote: > > > > > > > > 1. The logical type makes the implementation that assumes all arrays > are > > > fixed length, on par in performance with Option B > > > > 2. Without the logical type (and with a little bit extra complexity) > a > > > smart enough *reader* can walk the def/rep levels before decoding, > infer > > > (1) - the writer wrote fixed len arrays - and call the implementation > in > > > (1). Compared to (1) this is 1.5x slower. > > > > > > Then nothing is stopping implementations from doing this today? Gunnar > > > already merged this. We should document it as a useful implementation > > > optimization. > > > > > > > > > I have been thinking about whether we could obtain Option B-like levels > > > without introducing a new repetition type. One possibility would be to > add > > > an optional i32 length field to SchemaElement, allowing us to represent > > > types such as: > > > > > > optional float embedding[3]; > > > > > > Here, optional controls the nullability of the entire vector, while > length > > > = 3 means that every present vector contains exactly three floats. The > > > fixed dimension would not contribute > > > a repetition level. The presence of length would itself carry the > > > fixed-vector semantics, so a separate VECTOR logical annotation may > not be > > > necessary. > > > > > > Given the following three vectors: > > > > > > [1.0, 2.0, 3.0] > > > null > > > [4.0, 5.0, 6.0] > > > > > > the physical representation would be: > > > > > > repetition levels: omitted > > > definition levels: [1, 0, 1] > > > physical values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] > > > > > > Reader would use SchemaElement.length=3 to interpret this. > VECTOR-unaware > > > readers must therefore reject the vector columns. One way to provide > that > > > gate would be to require VECTOR-specific page encodings, such as > > > VECTOR_PLAIN and VECTOR_BYTE_STREAM_SPLIT, on pages with such a > column. A > > > VECTOR-unaware reader would reject the unsupported encoding before > > > attempting to decode the page. > > > > > > The other possible gating mechanism is, of course, versioning. :) > > > > > > The primary purpose of a VECTOR encoding here would be to gate out > > > VECTOR-unaware readers cleanly. There may also be another motivation: > > > vector similarity workloads may benefit from storing values by > > > dimension rather than by vector. > > > > > > For M vectors of length N: > > > - By vector: v0[0] ... v0[N], v1[0] ... v1[N], ... > > > - By dimension: v0[0] ... vM[0], v0[1] ... vM[1], ... > > > > > > This layout is discussed in the PDX paper [1] and would naturally be > > > represented as a page encoding in parquet. > > > > > > Would an augmented Option B along these lines be worth exploring? > > > > > > Rok > > > > > > [1] https://arxiv.org/pdf/2503.04422 > > > > > > On Fri, Jul 10, 2026 at 7:02 PM Andrew McCormick via dev < > > > [email protected]> wrote: > > > > > > > Hi Antoine, > > > > > > > > I wrote some more code to answer your question, details here: > > > > https://termbin.com/0zzc8. > > > > > > > > I added a nullability knob to the benchmark that writes an optional > outer > > > > array with zero nulls actually present (your "empty lists need \ > > > > a def level" case, but at 100% presence) and verified on disk that > this > > > > really is maxDef=2 (asserted in a test). The results are: > > > > arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) > delta > > > > B/VECTOR 1401 +/- 6 1403 +/- 12 > +0.2% > > > > (same required file; no nullable form) > > > > C-hint 1436 +/- 8 1428 +/- 8 -0.6% > > > > (annotation-aware skip; FLAT) > > > > C-naive 2606 +/- 11 2603 +/- 10 -0.1% > > > > (full Dremel decode) > > > > LIST 2613 +/- 11 2617 +/- 13 +0.1% > > > > > > > > Note that the option B number above isn't actually possible with the > > > > optional array, since Rok's impl only kicks in on required arrays, > but I > > > > included it here for reference to show the optional array doesn't > slow > > > > option C down. > > > > > > > > > > > > On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> > > > wrote: > > > > > > > > > > > > > > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit : > > > > > > I built a prototype of hint-supported reads for option C on top > of > > > > Rok's > > > > > > work. Here's the results I see: > > > > > > > > > > > > arm ns/row (mean +/- sd) note > > > > > > A/FLBA 2730 +/- 13 no levels on disk > (FLBA->float > > > > > > reinterpret adds a bit) > > > > > > B/VECTOR 2337 +/- 10 no levels, not forward > > > compatible > > > > > > C-hint 2356 +/- 5 skip-levels reader on a > plain > > > > > > LIST; fully backward-compatible > > > > > > C-dremel 3830 +/- 22 annotation ignored, full > Dremel > > > > > > (aka what Rok measured) > > > > > > > > > > > > So basically when you use the hint C is within noise of B (<1%). > Full > > > > > > details and code here: https://termbin.com/kj2x > > > > > > (gist isn't availble on my db github). > > > > > > > > > > One remaining question is what happens for definition levels (not > > > > > repetition) in option C. Empty lists need a specific definition > level > > > to > > > > > encode, therefore option C makes the max def level larger than > option > > > B. > > > > > > > > > > An optional column with option C might therefore take more time > > > decoding > > > > > than option B (especially if crossing the threshold from 1-bit to > 2-bit > > > > > levels). > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected] > > > > > > wrote: > > > > > >>> > > > > > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit : > > > > > >>>>>> 2. Without the logical type (and with a little bit extra > > > > complexity) > > > > > >> a > > > > > >>>>>> smart enough *reader* can walk the def/rep levels before > > > decoding, > > > > > >> infer > > > > > >>>> > > > > > >>>>> At the cost of higher implementation complexity and > maintenance > > > > cost. > > > > > >>>>> Does any mainstream open source implementation of Parquet do > > > this? > > > > > >>>> > > > > > >>>> Triggered by the conversation on the call last week, I > implemented > > > > > >>>> pretty much this in Hardwood [1]. > > > > > >>> > > > > > >>> Great, thank you. `FixedSizeListDetector.java` is highly > > > non-trivial > > > > > and > > > > > >>> definitely has a maintenance cost. Though part of it seems > about > > > not > > > > > >>> having a RLE parser abstraction available. > > > > > >> > > > > > >> Yes, I think we all agree that a dedicated type will make > > > maintainers' > > > > > >> lives much easier and is the right solution eventually. But > until > > > that > > > > > >> has landed, I think there's some juice worth the squeeze here. > > > > > >> > > > > > >>> > > > > > >>> Regards > > > > > >>> > > > > > >>> Antoine. > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
