devmadhuu opened a new pull request, #10696:
URL: https://github.com/apache/ozone/pull/10696
## What changes were proposed in this pull request?
**Problem:**
Recon keeps its own copy of the OM DB and syncs it two ways: incremental
delta updates (`getDBUpdates`) for the normal case, and a full OM DB snapshot
download as the fallback when delta sync cannot be used (Recon fell too far
behind, OM no longer has the requested updates, etc.).
The full-snapshot fallback was the weak part:
- It used a single one-shot GET `/dbCheckpoint` call that streamed the
entire OM DB as one tar.
- Any mid-stream failure restarted the download from byte 0.
- It always transferred the complete DB, even the SST files Recon already
had.
On large clusters the OM DB is big, so this approach was slow and fragile.
**Approach:**
Instead of maintaining Recon's
own download logic, Recon now reuses the same incremental bootstrap mechanism
an OM follower uses to catch up from the OM leader:
- A chunked POST `/v2/dbCheckpoint `request that carries a
`toExcludeList[]` of SST files the caller already has.
- The leader only sends the missing files and hard-links the rest (dedup),
ending the stream with a completion marker.
- Resume/retry behavior comes from the shared, proven OM code path rather
than a Recon-specific re-implementation.
On top of the follower behavior, Recon seeds the exclude list from its own
currently-installed (live) OM DB before the download starts. RocksDB SST file
numbers are content-stable within a DB lineage, so a `000123.sst `Recon
already has is byte-identical to the leader's. Those SSTs are hard-linked into
the candidate dir and advertised in the exclude list, so even a "full" snapshot
only transfers the SST files that actually changed. The assembled DB is always
complete and correct — a full snapshot still triggers a full task reprocess in
Recon, exactly as before; only the bytes on the wire go down.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15766
## How was this patch tested?
**Testing:**
- New unit tests `TestReconRDBSnapshotProvider`
- Updated `TestOzoneManagerServiceProviderImpl`— drives the new provider
path for success and failure cases.
- Existing `TestReconWithOzoneManager.testOmDBSyncWithSeqNumberMismatch
`provides end-to-end coverage of the full-snapshot fallback against a
mini-cluster
- `TestReconWithOzoneManagerHA` updated to drop the stale v1-URL assertion,
keeping its end-to-end leader-sync verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
