SaketaChalamchala opened a new pull request, #11428: URL: https://github.com/apache/ozone/pull/11428
## What changes were proposed in this pull request? Developed with the help of Cursor AI. This PR is intended as the foundation for a more efficient snapshot diff ([HDDS-9154](https://issues.apache.org/jira/browse/HDDS-9154)). This PR needs to be merged after https://github.com/apache/ozone/pull/11296 Baseline DAG based diff uses the raw SST iterator to scan the delta set of SST files between from and to snapshots. The RocksDB keys scanned from the delta are then looked up in from and to snapshots using random point lookups to create diff candidate sets. These diff candidate sets from both sides are then compared to generate the diff report. The new `DagDiffSequentialReader` mirrors `FullDiffSequentialReader` in what it produces (`newList`, `oldList`, from/to reverse-edge indexes, and directory diff-candidate tracking) while sourcing the to-side from delta SST K-way merges instead of full table scans. Details: For file/key tables, 1. K-way merge over to-side delta SST files with sequence gating (`fromSnapshot.dbTxSequenceNumber`) delegated to `LatestVersionedKWayMergeIterator` to populate the `newList`. This eliminates the random lookups on the toSnapshot file/key tables. 2. Interleaved batching at `SnapDiffJobStore.getWriteBatchSize()`: delta keys are accumulated and looked up via `multiGetSkipCache` against the raw fromSnapshot table to populate `oldList` instead of individual random point lookups. 3. Operates on raw `Table<byte[], byte[]>` handles and persisted protobuf bytes via `SnapshotDiffValueParser` (no re-encoding through typed helpers). For directory tables (FSO only), 1. Similar to the fileTable scan K-way merge over to-side delta SST files with sequence gating is used to populate the `newList`. The diff candidates from this scan are tracked. 2. Full sequential scan of `fromSnapshot.directoryTable` builds `fromEdges` and `oldList` is populated for diff candidates tracked in 1. 3. Full sequential scan of `toSnapshot.directoryTable` builds `toEdges`. - **DagDiffSequentialReader** owns sequential scanning of delta files and tables. - **SnapDiffJobStore** owns per-job temporary RocksDB column families: ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15393 ## How was this patch tested? Unit Tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
