[
https://issues.apache.org/jira/browse/HDDS-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDDS-14983:
--------------------------------------
Assignee: Wei-Chiu Chuang
> [Docs] Snapshot diff lifecycle
> ------------------------------
>
> Key: HDDS-14983
> URL: https://issues.apache.org/jira/browse/HDDS-14983
> Project: Apache Ozone
> Issue Type: Improvement
> Components: documentation, Snapshot
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Attachments: Gemini_Generated_Image_yr8ptayr8ptayr8p.png
>
>
> The lifecycle of a Snapshot Diff in Ozone is an asynchronous, job-based
> process designed to handle potentially millions of key changes without
> blocking
> the Ozone Manager.
> Based on SnapshotDiffManager.java, here is the step-by-step lifecycle:
> 1. Initiation & Job Queuing (getSnapshotDiffReport)
> * Request: A user requests a diff between fromSnapshot and toSnapshot.
> * Job Key: A unique snapDiffJobKey is generated using the UUIDs of the two
> snapshots.
> * Check/Create Job:
> * If a job already exists for this pair, its current status is
> returned.
> * If no job exists, a new SnapshotDiffJob is created with status
> QUEUED and persisted in the snapDiffJobTable (RocksDB).
> * Submission: The job is submitted to the snapDiffExecutor (a thread
> pool). The status transitions from QUEUED to IN_PROGRESS.
> 2. Preparation & Snapshots Opening (generateSnapshotDiffReport)
> * Metrics: Increments the numSnapshotDiffJobs metric.
> * Snapshot Handles: The service acquires "Active Snapshot" handles
> (rcFromSnapshot, rcToSnapshot) which are reference-counted to prevent the
> snapshots
> from being deleted while the diff is running.
> * Temporary Tables: Three temporary RocksDB Column Families (CF) are
> created specifically for this Job ID:
> 1. fromSnapshotColumnFamily: Maps Object IDs to Key Names in the
> source.
> 2. toSnapshotColumnFamily: Maps Object IDs to Key Names in the target.
> 3. objectIDsColumnFamily: Tracks the union of unique Object IDs and
> whether they are directories.
> 3. Delta Discovery (getDeltaFilesAndDiffKeysToObjectIdToKeyMap)
> * Efficient Diff (Native): If enabled, it uses the RocksDBCheckpointDiffer
> to identify the exact SST files that changed between the two snapshot
> checkpoints.
> * Full Diff (Fallback): If native libs are missing or a full diff is
> forced, it performs a complete scan of the key tables.
> * SST Parsing: It streams keys from the delta SST files.
> * Object Mapping: It populates the temporary "Object ID to Key Name" maps.
> This is critical for detecting Renames (same Object ID, different Key Name).
> 4. Path Resolution (FSO specific)
> * FSO Layout: If the bucket is File System Optimized (FSO), keys are
> stored as ParentID/FileName.
> * Path Resolver: The FSODirectoryPathResolver is used to recursively
> resolve the ParentID into a full human-readable path (e.g.,
> vol1/bucket1/dir1/file1).
> 5. Report Generation (generateDiffReport)
> * Comparison: The service iterates through the union of all Object IDs
> found in the delta.
> * Classification: It classifies changes into four types:
> * CREATE: Present in toSnapshot map but not fromSnapshot.
> * DELETE: Present in fromSnapshot map but not toSnapshot.
> * RENAME: Present in both maps with the same Object ID but different
> paths/names.
> * MODIFY: Present in both with the same path, but metadata (like ACLs
> or Block Versions) has changed.
> * Persistence: The resulting DiffReportEntry objects are serialized and
> stored in the snapDiffReportTable.
> 6. Finalization
> * Status Update: On success, the job status in snapDiffJobTable is updated
> to DONE, and the totalDiffEntries count is saved.
> * Cleanup: The temporary Column Families used for Object ID mapping are
> dropped and closed. The snapshot reference counts are decremented.
> 7. Consumption & Pagination (createPageResponse)
> * Polling: The client polls the OM for the report.
> * Paged Results: Since reports can be massive, the OM returns them in
> pages.
> * Token: Each response includes a token (the index of the next entry). The
> client provides this token in the next call to fetch the subsequent page.
> 8. Garbage Collection (Background)
> * Cleanup Service: (Referenced as DiffCleanupService) After a configurable
> interval, a background task deletes the job from snapDiffJobTable and the
> associated report entries from snapDiffReportTable to reclaim space.
> Summary of Job Statuses:
> * QUEUED: Job registered, waiting for a thread.
> * IN_PROGRESS: Currently scanning SSTs or generating the report.
> * DONE: Report is ready for paging.
> * FAILED: An error occurred (e.g., I/O error); includes a reason.
> * REJECTED: Too many concurrent jobs; the client should retry later.
> * CANCELLED: The job was manually stopped via a cancel request.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
