SaketaChalamchala opened a new pull request, #11083: URL: https://github.com/apache/ozone/pull/11083
## What changes were proposed in this pull request? Developed with the help of Cursor AI. This PR is intended as the foundation for a more efficient snapshot diff ([HDDS-9154](https://issues.apache.org/jira/browse/HDDS-9154)). Baseline full diff performs random reads against snapshot DBs and holds large in-memory maps. The optimized design replaces that with: 1. Sequential table scans over both snapshots 2. Persisted per-job intermediate lists keyed by objectId 3. Update-id gating (OM HA) to limit compare-signature work to likely-changed objects 4. Deferred path resolution via FSO edge column families - **FullDiffSequentialReader** scans raw snapshot tables (`Table<byte[], byte[]>`) so `SnapshotDiffValueParser` operates on exact persisted protobuf bytes (no decode/re-encode round trip). Scan order: file/key tables first, then directory tables (FSO). Each table pair runs to-side, then from-side. - **HA gating:** admits rows with `updateID > fromSnapshotDbTxSequenceNumber`. Rows with missing, zero, or `DEFAULT_OM_UPDATE_ID (-1)` are always candidates (conservative fallback). - **SnapDiffJobStore** owns per-job temporary RocksDB column families: - `{jobId}-new-list`, `{jobId}-old-list` — keyed by objectId - `{jobId}-to-edges`, `{jobId}-from-edges` — FSO directory edges (`parentId`, `objectId`) → `name` - `{jobId}-cand-ids` — spill target when the in-memory diff-candidate set exceeds the configured limit - **EntryValue** — compact fixed-layout (`parentId`, `name`, `isDir`, `signature`) (shared with the future DAG diff sequential reader). - **Config:** `ozone.om.snapshot.diff.max.in.memory.entries.per.job` (default 1M) bounds the in-memory diff-candidate set before spill to RocksDB. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15394 (Please replace this section with the link to the Apache JIRA) ## How was this patch tested? Unit Tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
