I finally got to reading your blog post. Thanks for sharing. The interleaved columns trick literally blew my mind :)
Some of those could be made first class in the project. - Aligning pages on key boundaries when sorting could be a writer option. Probably with a minimum size or something. - Making pages splittable (zstd frames) could be first class in the page header. Especially with new encodings where we can decode one value independently (like FSST and ALP). The writer would pay attention to align boundaries with values. - It seems that an analytics engine could reduce the IO just by being aware of column interleaving (the column chunks overlap). Or we just allow pages for a column to not be contiguous (per Dan's proposal). On Sat, Aug 1, 2026 at 7:31 PM Haocheng Liu <[email protected]> wrote: > Thanks for the sharing here as I’m also tackling similar random access > improvement to Parquet for my firm’s AI use cases. Great blog. > > I personally also learnt and benefited a lot from Weston Pace’s blog > series[1] on how lance evolved their file format. There must be a lot of > opportunities for the two communities to advance this area. > > Best > Haocheng > > [1] > > https://www.lancedb.com/blog/file-readers-in-depth-parallelism-without-row-groups > < > https://www.lancedb.com/blog/file-readers-in-depth-parallelism-without-row-groups > > > > > On Fri, Jul 31, 2026 at 4:41 AM Will Edwards via dev < > [email protected]> > wrote: > > > Howdy folks :) > > > > At Spotify we have been exploring how to efficiently use the data lake - > > something normally organised and optimised for big batch workloads - for > > fast point queries too. Eg the AI agent that needs to answer your > question > > about what you did last summer. > > > > We found that if we extract metadata and put it in a fast key value store > > we can know exactly which parts of which files to read without doing the > > normal loading footers and searching, which dramatically changes the > > performance and cost profile. And that there are some tricks you can do > > when writing parquet files that helps this access patterns. It’s > basically > > not so different from the metadata store that speeds up analytic > workloads > > but by key instead. > > > > Here’s blog we just posted that is hopefully interesting: > > > > > > > https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries > > > > Happy to go into more details and share perf tips and tricks if you have > > any thoughts or just wanna chat about Parquet perf! > > > > Best, > > Will > > > > > -- > Regards, > Haocheng >
