+1 Great to see this. Happy to engage as reviewer on RFC.
We are working on a highly-scalable vector search index for Hudi. It would be great to support syncs from other formats. On 2026/09/08 20:19:00 Vinish Reddy wrote: > Hi everyone, > > I opened RFC-4 as a pull request for review. > https://github.com/apache/incubator-xtable/pull/933 > > The RFC follows the earlier indexing thread [1] and the feature request > #887 [2]. > > What the RFC proposes: > > - A canonical index model for XTable: a record-level (primary) index and a > secondary index first, with expression and vector indexes left for later > work. > - The Hudi metadata table as the initial storage for these indexes, so > Iceberg, Delta Lake, Paimon and Parquet tables get indexes today. Iceberg > has a secondary index design in progress but no index support in the > specification yet. When Iceberg defines an index format, XTable can convert > indexes between formats the way XTable converts table metadata today. > - A small lookup API that returns the file and the row position of each > matching row, so an engine can accelerate joins and merges instead of > scanning the target table. > - Index builds reuse the existing sync loop. Record-level index builds need > a distributed engine, so they run behind an execution provider interface, > and xtable-core stays runnable without Spark. > > The RFC has three Mermaid flowcharts, and the plain diff shows only the > source of each one. Two ways to see the rendered diagrams: > > - Open the rendered file directly: > https://github.com/vinishjail97/onetable/blob/rfc-4-index-support/rfc/rfc-4/rfc-4.md > > - Or open the Files changed tab on the pull request and click the "Display > the rich diff" button on the file header. > > I would like feedback from the community on the index model, the lookup > API, the phasing and anything the RFC misses. Please comment on the pull > request or reply on this thread. > > Thanks, > Vinish > > [1] https://lists.apache.org/thread/kb3p85rgy0qkvjgdgn3580h36y1qov8c > [2] https://github.com/apache/incubator-xtable/issues/887 >
