chrevanthreddy opened a new issue, #19501: URL: https://github.com/apache/hudi/issues/19501
Parent: #19094 Related: #18676 ## Summary Implement Tier-1 maintenance for the vector metadata partition: fold accumulated deltas into packed posting blocks, repair locators, and checkpoint freshness metadata. This pass bounds read amplification without changing centroid geometry. It is deliberately separate from LIRE split/merge maintenance. ## Scope - Trigger maintenance from additive `deltaCount` and `tombstoneCount` thresholds. - Merge posting deltas with surviving block contents per `(generation, cluster, shard)`. - Join against the authoritative record-level index to remove dead or superseded postings. - Repack survivors while preserving `(fileGroupId, rowPosition)` ordering. - Reset delta/tombstone counters and update live statistics. - Refresh locators for rewritten base-table file groups. - Resolve MoR `rowPosition = -1` entries where physical positions become available. - Advance a verified frontier checkpoint and prune markers only when checkpoint evidence makes pruning safe. - Optionally produce an invalidation bitmap from this same evidence-gated pass; it is an optimization, not a correctness dependency. ## Correctness requirements - Compaction cannot resurrect tombstoned or superseded postings. - Repacked blocks preserve query and exact-rerank semantics. - Marker pruning cannot destroy the only proof needed to verify frontier continuity. - No centroid moves or cluster split/merge occurs in Tier 1. ## Acceptance criteria - Read amplification remains bounded after sustained mutation. - Tests cover mixed blocks/deltas/tombstones, cluster-changing updates, locator rewrites, MoR locator refinement, counter reset, checkpoint advancement, and safe marker pruning. - Repacking is reusable as the write primitive for later LIRE maintenance. ## Dependencies - Incremental vector-index update hook and additive statistics. - Freshness marker/frontier contract. - RLI arbitration and authoritative liveness checks. ## Blocks - LIRE split/merge maintenance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
