+1 on overall idea. I frequently talk to users, who want to keep Hudi indexes on queries and still be writing Delta Lake or Iceberg.
We can start there in a flexible way and evolve as different formats add similar capabilities? I think this will at-least have 12-18 months of shelf life. On 2026/07/01 00:39:24 Vinish Reddy Pannala wrote: > Following up on my earlier note - a few links didn't make it into the > original message, so here are the right ones for reference. > > Hudi's indexing overview. > 1. Docs: https://hudi.apache.org/docs/indexes/ > 2. The index implementations in the Hudi source - expression, record, and > secondary indexes: > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/index > > 3. The metadata-table index partitions (files, column_stats, bloom_filter, > record_index, secondary_index, partition_stats): > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/metadata > > > Thanks, > Vinish > > On Tue, Jun 30, 2026 05:19 PM, Vinish Reddy Pannala < > [email protected]> wrote: > > > Hi all, > > > > Wanted to float an idea and get people's thoughts. > > > > Right now Apache XTable(Incubating) translates table metadata so one copy > > of data can be read across Hudi, Iceberg and Delta. I believe a good > > addition for the project is to help engines query that data more > > efficiently by building indexes. > > > > Apache Hudi has a good writeup on why indexes matter - column stats, bloom > > filters, record-level, expression, secondary, and vector indexes. The short > > version is that without them, engines end up scanning far more data than > > they need to. > > > > Since XTable already reads the file listing and Parquet metadata during > > conversion, it seems well placed to build indexes from that same > > information and expose them into each format's native index mechanism. That > > would help both structured workloads (pruning, point lookups) and > > unstructured/vector ones (similarity search over embeddings for AI/RAG use > > cases). > > > > Before going further I wanted to ask the community: > > > > - Is this something the community sees value in? > > - Which indexes would be most useful to start with? > > - Any interest in vector indexes specifically, given where AI workloads > > are heading? > > > > Would love to hear thoughts. > > > > Thanks, > > Vinish > > > > >
