+1 from my side too. We can get the design discussion started in our sync calls.
On Wed, Jul 1, 2026 at 10:54 AM Vinoth Chandar <[email protected]> wrote: > +1 on overall idea. > > I frequently talk to users, who want to keep Hudi indexes on queries and > still be writing Delta Lake or Iceberg. > > We can start there in a flexible way and evolve as different formats add > similar capabilities? I think this will at-least have 12-18 months of shelf > life. > > On 2026/07/01 00:39:24 Vinish Reddy Pannala wrote: > > Following up on my earlier note - a few links didn't make it into the > > original message, so here are the right ones for reference. > > > > Hudi's indexing overview. > > 1. Docs: https://hudi.apache.org/docs/indexes/ > > 2. The index implementations in the Hudi source - expression, record, and > > secondary indexes: > > > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/index > > > > 3. The metadata-table index partitions (files, column_stats, > bloom_filter, > > record_index, secondary_index, partition_stats): > > > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/metadata > > > > > > Thanks, > > Vinish > > > > On Tue, Jun 30, 2026 05:19 PM, Vinish Reddy Pannala < > > [email protected]> wrote: > > > > > Hi all, > > > > > > Wanted to float an idea and get people's thoughts. > > > > > > Right now Apache XTable(Incubating) translates table metadata so one > copy > > > of data can be read across Hudi, Iceberg and Delta. I believe a good > > > addition for the project is to help engines query that data more > > > efficiently by building indexes. > > > > > > Apache Hudi has a good writeup on why indexes matter - column stats, > bloom > > > filters, record-level, expression, secondary, and vector indexes. The > short > > > version is that without them, engines end up scanning far more data > than > > > they need to. > > > > > > Since XTable already reads the file listing and Parquet metadata during > > > conversion, it seems well placed to build indexes from that same > > > information and expose them into each format's native index mechanism. > That > > > would help both structured workloads (pruning, point lookups) and > > > unstructured/vector ones (similarity search over embeddings for AI/RAG > use > > > cases). > > > > > > Before going further I wanted to ask the community: > > > > > > - Is this something the community sees value in? > > > - Which indexes would be most useful to start with? > > > - Any interest in vector indexes specifically, given where AI > workloads > > > are heading? > > > > > > Would love to hear thoughts. > > > > > > Thanks, > > > Vinish > > > > > > > > >
