nsivabalan commented on issue #20650:
URL: https://github.com/apache/hudi/issues/20650#issuecomment-6860215221
Thanks @vamshipasunuru1 for the detailed report, and @danny0405 for the
reconstruction proposal and comparison. Here's another option for discussion.
It has the same correctness property as the archived-prefix reconstruction
(accept a mismatch only when archival is known to explain it), but needs no
archive I/O and no change to the request protocol.
**Idea: the timeline server remembers its previous snapshot and how it moved
on from it**
On every view reload, the server compares the timeline it had before with
the one it has after, and records one previous snapshot:
- `prevHash`: hash of the timeline before the reload.
- `archivalOnly`: true when the new timeline is exactly the old one with a
prefix of the oldest instants removed, plus at most one trailing clean.
"Exactly" means the remaining instants match one for one on (requestedTime,
action, state): `hash(prev.instantsFrom(newFirst) [+ trailing clean]) ==
hash(new)`.
Any other kind of reload (new commit, late completion, compaction or
clustering state change, restore/rollback) sets `archivalOnly = false`.
A request is served without a reload when the client hash matches the
current timeline (today's behaviour), **or** when `clientHash == prevHash &&
archivalOnly`.
```text
gen1 (prev): [T50 ............ T100] hash H1
gen2 (cur): [T60 ...... T100, clean101] hash H2
gen2 == gen1 minus [T50, T60) + trailing clean → archivalOnly
= true
stale client sends H1 → H1 == prevHash && archivalOnly → serve from gen2,
no reload
fresh client sends H2 → matches current, as today
```
**Why this stays safe**
- Late completions are still detected. If a commit or replacecommit
requested inside the overlap completed between the two snapshots, the overlap
differs and `archivalOnly` is false. Because instants are ordered by requested
time, a hash over only the latest N instants (#20651) cannot see these, which
is how the server could end up returning replaced or stale file groups.
- Anything that moves the end of the timeline (new commits, restore)
disqualifies the shortcut. A client that is genuinely behind still gets today's
400 and falls back to its local view.
- Serving the client from gen2 is equivalent because archival keeps a
replacecommit until its replaced file groups are cleaned, and never archives
pending compactions. We should state this rule in the code and test it.
**Request protocol and compatibility**
The hash already identifies the client's full instant list, so no new
request parameters are needed. `timelinehash` and `lastinstantts` keep their
meaning, and older clients benefit without changes. The pre-handler check
(`syncIfLocalViewBehind`) and the final check (`isLocalViewBehind`) must use
the same predicate and read `prevHash`/`archivalOnly` under the view lock, so a
reload between them can't mix generations.
**The cold-start case in this issue**
In the reported sequence, the embedded server loads its view for the first
time *after* archival, so there is no previous snapshot to compare against. To
cover it, the driver registers its write-start timeline with the embedded
server (same JVM, so this only passes over the instant list and hash, with no
extra storage reads), and the server records it as the previous snapshot. Later
reloads follow the rules above. A standalone server that never saw the client's
snapshot keeps today's behaviour. The archived-prefix reconstruction could be
added later as a fallback for that case if needed.
**Driver restarts**
The previous snapshot is held only in memory and is never persisted, so a
restart can't bring back a stale or wrong snapshot. When the driver restarts,
the embedded timeline server restarts with it, and its state starts empty. The
new driver loads a fresh timeline and registers it as the write-start snapshot,
so the server and the new executors start from the same timeline. Executors
from the old driver can't reach the new server (it has a new host/port), and
their tasks are re-run with the new driver's timeline. If a standalone timeline
server restarts, it also starts empty: clients holding older hashes get today's
behaviour (reload, then 400 and local fallback), not a wrong answer. In both
cases, losing the state only costs the optimization, never correctness.
**Scope**
- Only one previous snapshot is kept. This covers one archival round between
the client's snapshot and the server's reload, which is the common case for a
write that lasts minutes. It can be extended to a short chain later if
long-running writes need it.
- Combine it with #20239, which stops repeated reloads for whatever still
falls back (genuine server-ahead cases), so the remaining cases also don't
stall the stage.
**Tests (real files, not mocks)**
1. Archival and a trailing clean before the server's first load, with the
snapshot registered: requests are served remotely, with no repeated syncs and
correct base files.
2. A replacecommit requested inside the overlap completes between snapshots:
the server reloads and does not return replaced file groups.
3. A late-completing commit (the `001/002/clean003` case from
`testRealViewRefreshesOutOfOrderCompletion`): the server reloads and returns
the base file from `002`.
4. A new commit or restore after the previous snapshot: the shortcut is not
taken and existing behaviour is preserved.
5. A reload between the pre-handler and final checks: no request is accepted
against a mix of generations.
6. Driver restart during a write (embedded server restarted with empty
state): the re-run tasks are served remotely against the new driver's
registered snapshot, and no request is accepted against the old driver's
snapshot.
Happy to discuss here or on a call before anyone starts on an implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]