Hi all, Wanted to float an idea and get people's thoughts.
Right now Apache XTable(Incubating) translates table metadata so one copy of data can be read across Hudi, Iceberg and Delta. I believe a good addition for the project is to help engines query that data more efficiently by building indexes. Apache Hudi has a good writeup on why indexes matter - column stats, bloom filters, record-level, expression, secondary, and vector indexes. The short version is that without them, engines end up scanning far more data than they need to. Since XTable already reads the file listing and Parquet metadata during conversion, it seems well placed to build indexes from that same information and expose them into each format's native index mechanism. That would help both structured workloads (pruning, point lookups) and unstructured/vector ones (similarity search over embeddings for AI/RAG use cases). Before going further I wanted to ask the community: - Is this something the community sees value in? - Which indexes would be most useful to start with? - Any interest in vector indexes specifically, given where AI workloads are heading? Would love to hear thoughts. Thanks, Vinish
