danny0405 commented on issue #20650:
URL: https://github.com/apache/hudi/issues/20650#issuecomment-6847315815

   A possible fix is to reconstruct the missing archived prefix on the server 
and keep the existing full-timeline hash comparison.
   
   **Proposed protocol and comparison**
   
   1. Add `firstinstantts` to remote-view requests, taken from the same 
captured, filtered client timeline used to compute `timelinehash`. Keep the 
existing full hash and `lastinstantts`.
   2. Keep the existing exact-match and trailing-clean fast paths. If they fail 
and the client's first instant precedes the server's first instant, with 
overlap remaining in the active timeline, load archived instant identities in 
the half-open range `[clientFirstInstant, serverFirstInstant)`.
   3. Prepend those archived instants to the server's active timeline for 
comparison only, using the same ordering, filtering, and instant representation 
as the full-hash check.
   4. If the reconstructed timeline's hash matches the client's hash, 
optionally applying the existing single-trailing-clean exception, serve the 
request remotely without syncing the view. Otherwise, retain the existing 
sync-and-validation path.
   
   For example:
   
   ```text
   Client snapshot:          [001, 002, 003]             hash H
   Server active timeline:        [002, 003, clean004]
   Archived prefix loaded:   [001]
   
   Reconstructed comparison: [001, 002, 003, clean004]
   After clean exception:    [001, 002, 003]             hash H -> serve 
remotely
   ```
   
   The reconstructed timeline is only used to establish equivalence; it does 
not replace the filesystem view's active timeline. In particular, do not 
truncate the server's tail at the client's last instant to force a match. Extra 
data commits or action/state differences should continue to prevent acceptance.
   
   Compared with hashing only the latest N instants in #20651, this retains 
checking of the discarded prefix: an older instant that completed after the 
client captured its timeline would appear in the reconstructed history and 
prevent a match. It also needs no client/server negotiation round trip or 
choice of N.
   
   **Scope and performance**
   
   - Hudi already has range-filtered archive loading and `LoadMode.ACTION`, so 
this should read instant identities rather than full commit metadata.
   - Cache successful equivalence checks for the client snapshot and the 
particular server view generation. Invalidate on view changes/recreation or 
explicit refresh, and use the same equivalence predicate in the post-handler 
consistency check. Concurrent requests sharing a snapshot should share the 
archive lookup.
   - Preserve existing behavior when the new parameter is absent or archive 
evidence is unavailable/incomplete. Initially scope this to ordinary prefix 
archival with active-timeline overlap; timelines with holes and entirely 
archived client snapshots can retain the existing path.
   
   This is a design proposal, not an implemented or tested fix. Validation 
should include real-file archival before lazy server initialization, archival 
plus a trailing clean, repeated concurrent requests returning correct remote 
results without fallback or repeated sync/archive reads, late completion, 
compaction/clustering transitions, and restore/deleted-instant recovery across 
the supported archive layouts.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to