+1 thanks to the integration of the Parquet format (by me).
On Sun, 5 Jul 2026, 17:03 Vaibhav Kumar, <[email protected]> wrote: > +1 from my side too. > > We can get the design discussion started in our sync calls. > > On Wed, Jul 1, 2026 at 10:54 AM Vinoth Chandar <[email protected]> wrote: > > > +1 on overall idea. > > > > I frequently talk to users, who want to keep Hudi indexes on queries and > > still be writing Delta Lake or Iceberg. > > > > We can start there in a flexible way and evolve as different formats add > > similar capabilities? I think this will at-least have 12-18 months of > shelf > > life. > > > > On 2026/07/01 00:39:24 Vinish Reddy Pannala wrote: > > > Following up on my earlier note - a few links didn't make it into the > > > original message, so here are the right ones for reference. > > > > > > Hudi's indexing overview. > > > 1. Docs: https://hudi.apache.org/docs/indexes/ > > > 2. The index implementations in the Hudi source - expression, record, > and > > > secondary indexes: > > > > > > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/index > > > > > > 3. The metadata-table index partitions (files, column_stats, > > bloom_filter, > > > record_index, secondary_index, partition_stats): > > > > > > https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/metadata > > > > > > > > > Thanks, > > > Vinish > > > > > > On Tue, Jun 30, 2026 05:19 PM, Vinish Reddy Pannala < > > > [email protected]> wrote: > > > > > > > Hi all, > > > > > > > > Wanted to float an idea and get people's thoughts. > > > > > > > > Right now Apache XTable(Incubating) translates table metadata so one > > copy > > > > of data can be read across Hudi, Iceberg and Delta. I believe a good > > > > addition for the project is to help engines query that data more > > > > efficiently by building indexes. > > > > > > > > Apache Hudi has a good writeup on why indexes matter - column stats, > > bloom > > > > filters, record-level, expression, secondary, and vector indexes. The > > short > > > > version is that without them, engines end up scanning far more data > > than > > > > they need to. > > > > > > > > Since XTable already reads the file listing and Parquet metadata > during > > > > conversion, it seems well placed to build indexes from that same > > > > information and expose them into each format's native index > mechanism. > > That > > > > would help both structured workloads (pruning, point lookups) and > > > > unstructured/vector ones (similarity search over embeddings for > AI/RAG > > use > > > > cases). > > > > > > > > Before going further I wanted to ask the community: > > > > > > > > - Is this something the community sees value in? > > > > - Which indexes would be most useful to start with? > > > > - Any interest in vector indexes specifically, given where AI > > workloads > > > > are heading? > > > > > > > > Would love to hear thoughts. > > > > > > > > Thanks, > > > > Vinish > > > > > > > > > > > > > >
