+1 Thanks for sharing the RFC Vinish, I have posted a few points in the rfc.Please take a look.
On Thu, Sep 10, 2026 at 2:11 AM Vinoth Chandar <[email protected]> wrote: > +1 > > Great to see this. Happy to engage as reviewer on RFC. > > We are working on a highly-scalable vector search index for Hudi. It would > be great to support syncs from other formats. > > On 2026/09/08 20:19:00 Vinish Reddy wrote: > > Hi everyone, > > > > I opened RFC-4 as a pull request for review. > > https://github.com/apache/incubator-xtable/pull/933 > > > > The RFC follows the earlier indexing thread [1] and the feature request > > #887 [2]. > > > > What the RFC proposes: > > > > - A canonical index model for XTable: a record-level (primary) index and > a > > secondary index first, with expression and vector indexes left for later > > work. > > - The Hudi metadata table as the initial storage for these indexes, so > > Iceberg, Delta Lake, Paimon and Parquet tables get indexes today. Iceberg > > has a secondary index design in progress but no index support in the > > specification yet. When Iceberg defines an index format, XTable can > convert > > indexes between formats the way XTable converts table metadata today. > > - A small lookup API that returns the file and the row position of each > > matching row, so an engine can accelerate joins and merges instead of > > scanning the target table. > > - Index builds reuse the existing sync loop. Record-level index builds > need > > a distributed engine, so they run behind an execution provider interface, > > and xtable-core stays runnable without Spark. > > > > The RFC has three Mermaid flowcharts, and the plain diff shows only the > > source of each one. Two ways to see the rendered diagrams: > > > > - Open the rendered file directly: > > > https://github.com/vinishjail97/onetable/blob/rfc-4-index-support/rfc/rfc-4/rfc-4.md > > > > - Or open the Files changed tab on the pull request and click the > "Display > > the rich diff" button on the file header. > > > > I would like feedback from the community on the index model, the lookup > > API, the phasing and anything the RFC misses. Please comment on the pull > > request or reply on this thread. > > > > Thanks, > > Vinish > > > > [1] https://lists.apache.org/thread/kb3p85rgy0qkvjgdgn3580h36y1qov8c > > [2] https://github.com/apache/incubator-xtable/issues/887 > > >
