zhaoyudi-creator commented on issue #19360: URL: https://github.com/apache/hudi/issues/19360#issuecomment-5099758610
**Case analysis of the truncated-hash approach** (covering async table services and multi-writer) Notation: `Lc` = lastInstant reported by the client, `Ls` = server's lastInstant. Rule: when the full hash differs, truncate the server timeline to `Lc` and re-hash — compare `H(server≤Lc)` against `H(client)`. Correctness red-line: **server-behind-but-skip = data error**; server-not-behind-but-reload = perf cost only. | Server state vs. client | Scenarios covered (incl. async table service / multi-writer) | Truncated hash | Decision | Outcome | |---|---|---|---|---| | **Server ahead or equal** (only holds instants `>Lc` beyond the client) | full hash already equal; extra `>Lc` new commit / clean / compaction / rollback / restore; multi-writer where the client lags and `server ⊇ client` | equal (falls back to `containsOrBeforeTimelineStarts` when fully equal) | **skip** | ✓ correct, **optimization kicks in**; the server already applied these instants into its view, and skipping does not undo any completed deletion/rollback — the client still reads a fresh, correct view through the server | | **Server behind** (missing some client-known instant within `≤Lc`) | normal: client has a newer commit; **async out-of-order**: server misses an *earlier* intermediate instant yet has a newer `Ls` (the #6179 case where naive latest-commit `containsOrBeforeTimelineStarts(Ls)=true` would wrongly skip); missing a client-known clean/rollback; multi-writer interleaving gap | differs | **reload** | ✓ correct, **never misses**; core invariant: `hash equal ⟺ server ⊇ client within ≤Lc`, so any genuinely-behind state lands here | | **Server not behind, but truncated hash still differs** | out-of-order making the server hold instants `≤Lc` the client hasn't seen yet; state transition (compaction pending→commit); `Lc=INVALID` (empty client view); archival lag; pending `LOG_COMPACTION` causing a filter-scope mismatch | differs | reload | ⚠️ one extra reload (perf only), **no correctness issue**, no worse than today | **Conclusion:** the first two rows cover every correctness-relevant case — whenever the server is genuinely behind (including async out-of-order and multi-writer interleaving) it always reloads and never misses; the optimization skips only when the server is truly ahead (holds only instants `>Lc`). The third row is a conservative fallback that costs performance but not correctness. Overall correctness is no weaker than the current full-hash implementation; the only added assumption is SHA collision-freedom, which the existing hash mechanism already relies on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
