Following up on my earlier note - a few links didn't make it into the original message, so here are the right ones for reference.
Hudi's indexing overview. 1. Docs: https://hudi.apache.org/docs/indexes/ 2. The index implementations in the Hudi source - expression, record, and secondary indexes: https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/index 3. The metadata-table index partitions (files, column_stats, bloom_filter, record_index, secondary_index, partition_stats): https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/metadata Thanks, Vinish On Tue, Jun 30, 2026 05:19 PM, Vinish Reddy Pannala < [email protected]> wrote: > Hi all, > > Wanted to float an idea and get people's thoughts. > > Right now Apache XTable(Incubating) translates table metadata so one copy > of data can be read across Hudi, Iceberg and Delta. I believe a good > addition for the project is to help engines query that data more > efficiently by building indexes. > > Apache Hudi has a good writeup on why indexes matter - column stats, bloom > filters, record-level, expression, secondary, and vector indexes. The short > version is that without them, engines end up scanning far more data than > they need to. > > Since XTable already reads the file listing and Parquet metadata during > conversion, it seems well placed to build indexes from that same > information and expose them into each format's native index mechanism. That > would help both structured workloads (pruning, point lookups) and > unstructured/vector ones (similarity search over embeddings for AI/RAG use > cases). > > Before going further I wanted to ask the community: > > - Is this something the community sees value in? > - Which indexes would be most useful to start with? > - Any interest in vector indexes specifically, given where AI workloads > are heading? > > Would love to hear thoughts. > > Thanks, > Vinish > >
