chrevanthreddy opened a new issue, #19499: URL: https://github.com/apache/hudi/issues/19499
Parent: #19094 Related: #18676 ## Summary Implement the source-instant marker, verified contiguous frontier, planner policy, and catch-up replay contract for MDT-backed vector indexes. The frontier is proof of contiguous source-timeline coverage, not merely the greatest instant observed. Missing coverage must never be mistaken for freshness. ## Scope - Add source-instant marker records keyed by index identity and source instant. - Track a verified contiguous frontier/watermark in the generation manifest. - Advance the frontier only after proving every required source instant is covered. - Handle archived source instants when verifying continuity. - Check index freshness during planning and apply configured `FAIL`, `WARN`, or exact-fallback behavior. - Report marker-frontier lag as `indexLagInstants`. - Implement catch-up replay using the same written-file reread and delta encoder as normal `buildUpdate`. - Atomically write replayed deltas/statistics and their original source-instant markers. - Detect and safely resume partial or interrupted replay. ## Correctness requirements - A marker gap prevents frontier advancement beyond the gap. - Maximum observed marker is never treated as contiguous coverage. - No-op source commits are represented. - Archived timeline segments do not permit unverified frontier jumps. - Replay is idempotent and preserves source-instant identity. ## Acceptance criteria - Planner tests cover fresh, stale, missing, and policy-specific behavior. - Frontier tests cover gaps, no-op commits, archived instants, and interrupted replay. - Catch-up shares the normal update implementation rather than introducing a second encoder. - Metrics expose frontier lag and replay progress. ## Dependencies - Schema/payload support in #19097 / PR #19317. - Incremental vector-index update hook. ## Blocks - Generation activation after rebuild. - Marker pruning/checkpointing during Tier-1 compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
