chrevanthreddy opened a new issue, #19498: URL: https://github.com/apache/hudi/issues/19498
Parent: #19094 Related: #18676 ## Summary Implement incremental maintenance for the MDT-backed vector index after each committed base-table write. This is the writer-side critical path for keeping an active vector generation current. It follows the existing `EXPRESSION_INDEX` written-file reread pattern rather than relying on incomplete write-status payloads. ## Scope - Ensure vector-index update dispatch reaches `VectorIndexer.buildUpdate` for every completed source instant, including commits with no changed vector rows. - Re-read files written by the source commit and extract authoritative vector rows and locators. - Load the ACTIVE generation's centroids and quantizer metadata once per update. - Assign and encode changed vectors against that ACTIVE generation. - Emit posting deltas and additive `ClusterStats` updates. - Emit the source-instant marker in the same MDT update as its posting/stat changes. - Represent MoR log-resident rows with `rowPosition = -1` until locator refresh. - Preserve idempotence for replay of an already-marked source instant. ## Correctness requirements - Update-in-place suppresses the old posting and exposes the replacement. - A vector-changing update that moves clusters cannot leave the old cluster posting visible. - Deletes do not remain visible through either approximate or exact-rerank search. - Marker publication is atomic with the corresponding vector-index changes. - A no-op source commit still produces marker coverage. ## Acceptance criteria - `VectorIndexer.buildUpdate` is no longer a no-op. - COW and MoR update paths are covered. - Posting deltas, additive statistics, and source markers are committed together. - Reprocessing is idempotent. - Unit/integration tests cover insert, update-in-place, cluster-changing update, delete, no-op commit, and MoR log-row location semantics. ## Dependencies - Schema/payload support in #19097 / PR #19317. - Read-side arbitration and continuation work in #19102 and #19103. ## Blocks - Freshness frontier and catch-up replay. - Generation rebuild catch-up and activation. - Tier-1 vector compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
