Howdy folks :) At Spotify we have been exploring how to efficiently use the data lake - something normally organised and optimised for big batch workloads - for fast point queries too. Eg the AI agent that needs to answer your question about what you did last summer.
We found that if we extract metadata and put it in a fast key value store we can know exactly which parts of which files to read without doing the normal loading footers and searching, which dramatically changes the performance and cost profile. And that there are some tricks you can do when writing parquet files that helps this access patterns. It’s basically not so different from the metadata store that speeds up analytic workloads but by key instead. Here’s blog we just posted that is hopefully interesting: https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries Happy to go into more details and share perf tips and tricks if you have any thoughts or just wanna chat about Parquet perf! Best, Will
