zhaoyudi-creator commented on issue #19360: URL: https://github.com/apache/hudi/issues/19360#issuecomment-5099729094
<p dir="ltr" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); color: rgb(10, 10, 10); font-family: "Anthropic Sans", system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 253, 252); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollba r-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-weight: 500;">Case analysis of the truncated-hash approach</strong><span> </span>(covering async table services and multi-writer)</p><p dir="ltr" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); color: rgb(10, 10, 10); font-family: "Anthropic Sans", system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 253, 252); text-decoration-thickness: initial; text-decoration-style: initial ; text-decoration-color: initial;">Notation:<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04); color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">Lc</code><span> </span>= lastInstant reported by the client,<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 10 7, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04); color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">Ls</code><span> </span>= server's lastInstant. Rule: when the full hash differs, compare<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04); color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">H(server≤Lc)</code><span> </span>against<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04); color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; l ine-height: 1.4;">H(client)</code><span> </span>— i.e. truncate the server timeline to<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04); color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">Lc</code><span> </span>and re-hash. Correctness red-line:<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-c olor: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-weight: 500;">server-behind-but-skip = data error</strong>; server-not-behind-but-reload = perf cost only.</p><div dir="ltr" class="overflow-x-auto" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); overflow-x: auto; color: rgb(10, 10, 10); font-family: "Anthropic Sans", system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 253, 252); text-decorati on-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> Server state vs. client | Scenarios covered (incl. async table service / multi-writer) | Truncated hash | Decision | Outcome -- | -- | -- | -- | -- Server ahead or equal (only holds instants >Lc beyond the client) | full hash already equal; extra >Lc new commit / clean / compaction / rollback / restore; multi-writer where the client lags and server ⊇ client | equal (falls back to containsOrBeforeTimelineStarts when fully equal) | skip | ✓ correct, optimization kicks in; the server already applied these instants into its view, and skipping does not undo any completed deletion/rollback — the client still reads a fresh, correct view through the server Server behind (missing some client-known instant within ≤Lc) | normal: client has a newer commit; async out-of-order: server misses an earlier intermediate instant yet has a newer Ls (the #6179 case where naive latest-commit containsOrBeforeTimelineStarts(Ls)=true would wrongly skip); missing a client-known clean/rollback; multi-writer interleaving gap | differs | reload | ✓ correct, never misses; core invariant: hash equal ⟺ server ⊇ client within ≤Lc, so any genuinely-behind state lands here Server not behind, but truncated hash still differs | out-of-order making the server hold instants ≤Lc the client hasn't seen yet; state transition (compaction pending→commit); Lc=INVALID (empty client view); archival lag; pending LOG_COMPACTION causing a filter-scope mismatch | differs | reload | ⚠️ one extra reload (perf only), no correctness issue, no worse than today </div><p dir="ltr" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); color: rgb(10, 10, 10); font-family: "Anthropic Sans", system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 253, 252); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); sc rollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-weight: 500;">Conclusion:</strong><span> </span>the first two rows cover every correctness-relevant case — whenever the server is genuinely behind (including async out-of-order and multi-writer interleaving) it always reloads and never misses; the optimization skips only when the server is truly ahead (holds only instants<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 107, 193); scrollbar-width: thin; scrollbar-color: rgba(31, 31, 30, 0.35) rgba(0, 0, 0, 0); font-family: "Anthropic Mono", ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 13px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(10, 10, 10, 0.04) ; color: rgb(10, 10, 10); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">>Lc</code>). The third row is a conservative fallback that costs performance but not correctness. Overall correctness is no weaker than the current full-hash implementation; the only added assumption is SHA collision-freedom, which the existing hash mechanism already relies on.</p> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
