An orthogonal discussion on the Iceberg dev ML: https://lists.apache.org/thread/xdkzllzt4p3tvcd3ft4t7jsvyvztr41j
It proposes to add index support to Iceberg. One use case is to leverage Parquet files to store inverted indexes. Best, Gang On Fri, Jul 18, 2025 at 1:32 AM Felipe Oliveira Carvalho < felipe...@gmail.com> wrote: > On Thu, Jul 17, 2025 at 8:18 AM Andrew Lamb <andrewlam...@gmail.com> > wrote: > > > > 1. Parquet file format seems have index page [1], but I don't know > who's > > > > The INDEX_PAGE type a fascinating point -- I am not sure what benefit > > writing indexes using that annotation would be 🤔 > > > > > Currently I don't know whether we can have some "offcial" sample index. > > > > I am not sure examples need to be "official" -- I suspect people would be > > interested in public open source examples of various types of indexes > that > > they could adapt to their own needs. > > > > Having at least a central location with namespaced index names and links to > the description of their implementation would help more libraries leverage > these indexes. Because the lack of standardization means only the libraries > that produced the files can leverage them. Which defeats the purpose of > using an open format somewhat. > > -- > Felipe > > > > Andrew > > > > On Wed, Jul 16, 2025 at 7:16 AM wish maple <maplewish...@gmail.com> > wrote: > > > > > Seems good. Personally I think > > > > > > 1. Parquet file format seems have index page [1], but I don't know > who's > > > using it. > > > 2. Currently, Parquet only have single column bloom filter and column > > > index. Maybe > > > some kind of multi-column or other filter might work > > > 3. Index can have different "levels", like Page Index is designed for > > > "Page", and bloom > > > filter / statistics for RowGroup. We can even define index for > "file" > > > > > > Currently I don't know whether we can have some "offcial" sample index. > > > Personally I > > > might be interested in some "sketches" > > > > > > Best, > > > Xuwei Fu > > > > > > [1] > > > > > > > > > https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L655 > > > > > > Andrew Lamb <andrewlam...@gmail.com> 于2025年7月16日周三 19:08写道： > > > > > > > I wrote a blog with Qi Zhu, Jigao Luo explaining how to embed user > > > defined > > > > indexes into Parquet files without needing any changes to the > > format[1]. > > > > > > > > I am sorry for the somewhat shameless self promotion, but I think > this > > > > topic may be of general interest to the community in the context of > > other > > > > extensions to the format we have discussed recently. Techniques such > as > > > > this widen potential usecases of Parquet without any need for > > consensus > > > or > > > > timeline for ecosystem adoption. > > > > > > > > Andrew > > > > > > > > [1]: > > > > > > > > > > https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/ > > > > > > > > > >
