vinishjail97 opened a new issue, #887: URL: https://github.com/apache/incubator-xtable/issues/887
### Feature Request / Improvement Umbrella issue for cross-format indexing in Apache XTable (incubating): building indexes during conversion and exposing them into each format's native index mechanism, so engines can prune and look up efficiently against XTable-written metadata. This tracks the outcome of the dev@ discussion, which had clear support but produced no tracking issue or RFC. Filing this so the work becomes actionable. ## The dev@ thread **[DISCUSS] Indexing support across table formats in Apache XTable(Incubating)** https://lists.apache.org/thread/kb3p85rgy0qkvjgdgn3580h36y1qov8c The proposal (2026-06-30): > Right now Apache XTable(Incubating) translates table metadata so one copy of data can be read across Hudi, Iceberg and Delta. I believe a good addition for the project is to help engines query that data more efficiently by building indexes. > > Since XTable already reads the file listing and Parquet metadata during conversion, it seems well placed to build indexes from that same information and expose them into each format's native index mechanism. That would help both structured workloads (pruning, point lookups) and unstructured/vector ones (similarity search over embeddings for AI/RAG use cases). Responses on the thread: **Vinoth Chandar** — start narrow, and expect a useful lifetime: > +1 on overall idea. > > I frequently talk to users, who want to keep Hudi indexes on queries and still be writing Delta Lake or Iceberg. > > We can start there in a flexible way and evolve as different formats add similar capabilities? I think this will at-least have 12-18 months of shelf life. **Rahil Chertara** — the format specs are heading this way, so conversion is the natural fit: > I feel that we will eventually have to support this as Iceberg is also planning on adding native indexing support within the format spec at some point. [1] So the natural step would be for xtable to handle this index conversion. **Vaibhav Kumar**: > +1 from my side too. > > We can get the design discussion started in our sync calls. **Sapienza** (author of the Parquet source integration): > +1 thanks to the integration of the Parquet format (by me). [1] Iceberg native indexing design doc, linked by Rahil: https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit ## Reference: Hudi's index surface Shared on the thread as the richest existing model to translate from: - Docs: https://hudi.apache.org/docs/indexes/ - Index implementations (expression, record, secondary): https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/index - Metadata-table index partitions (`files`, `column_stats`, `bloom_filter`, `record_index`, `secondary_index`, `partition_stats`): https://github.com/apache/hudi/tree/master/hudi-common/src/main/java/org/apache/hudi/metadata ## Proposed first slice Taking Vinoth's "start narrow" literally, rather than designing the whole surface first: **Statistics-based pruning, end to end.** Column stats and partition stats, sourced from the Parquet footers XTable already reads during conversion, written into each target's native mechanism. That choice is deliberate. It gives users a measurable query win, it exercises whatever internal representation we settle on, and it does not depend on resolving record-level or vector index design first. Everything else stays a follow-up under this umbrella: bloom filters, record-level index, expression index, secondary index, and vector indexes for similarity search. ## Design questions for the RFC An RFC should land under `rfc/` before the implementation, covering at least: 1. The internal representation of an index in the XTable model, and where it sits relative to the existing schema and column-stats extraction. 2. Which indexes we translate, and in which directions. Not every pairing is meaningful, and some are lossy. 3. Forward compatibility with the Iceberg spec work Rahil referenced, so we are not locked into a Hudi-shaped model when Iceberg lands native indexing. 4. Invalidation on incremental sync. An index that silently goes stale after a partial sync is worse than no index. 5. Cost and opt-out. Index building adds work to every sync, so it needs a config surface and a documented cost. 6. How this behaves for source formats with no index concept, for example the Parquet source. Note on numbering: `rfc/rfc-2/` in the repo is "Deletion Info Conversion" while PR #612 also proposes an RFC-2, so the number for this one should follow whatever #656 settles. ## Dependencies and related work - **#834**, Hudi table version 9. The Hudi 1.x metadata indexes only become usable for XTable-written tables once the Hudi target writes v9, so v9 gates the Hudi side of this. - **#832**, enable the MDT column-stats index for partitioned tables. - **PR #760**, column-statistics extraction with a Parquet footer fallback for Delta and Iceberg. Directly relevant to the first slice. - **PR #778**, column stats on by default for Hudi 1.1.x. - **PR #811**, option to disable column stats. Relevant to the cost and opt-out question. - **apache/hudi#19111**, which allows disabling the partition-stats index independently of column stats. Needed because partition-stats generation is not supported over externally created files. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
