I built a prototype of hint-supported reads for option C on top of Rok's work. Here's the results I see:
arm ns/row (mean +/- sd) note A/FLBA 2730 +/- 13 no levels on disk (FLBA->float reinterpret adds a bit) B/VECTOR 2337 +/- 10 no levels, not forward compatible C-hint 2356 +/- 5 skip-levels reader on a plain LIST; fully backward-compatible C-dremel 3830 +/- 22 annotation ignored, full Dremel (aka what Rok measured) So basically when you use the hint C is within noise of B (<1%). Full details and code here: https://termbin.com/kj2x (gist isn't availble on my db github). On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling <[email protected]> wrote: > On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]> wrote: > > > > Le 06/07/2026 à 23:29, Gunnar Morling a écrit : > > >>> 2. Without the logical type (and with a little bit extra complexity) > a > > >>> smart enough *reader* can walk the def/rep levels before decoding, > infer > > > > > >> At the cost of higher implementation complexity and maintenance cost. > > >> Does any mainstream open source implementation of Parquet do this? > > > > > > Triggered by the conversation on the call last week, I implemented > > > pretty much this in Hardwood [1]. > > > > Great, thank you. `FixedSizeListDetector.java` is highly non-trivial and > > definitely has a maintenance cost. Though part of it seems about not > > having a RLE parser abstraction available. > > Yes, I think we all agree that a dedicated type will make maintainers' > lives much easier and is the right solution eventually. But until that > has landed, I think there's some juice worth the squeeze here. > > > > > Regards > > > > Antoine. > > > > >
