smengcl opened a new pull request, #11575: URL: https://github.com/apache/ozone/pull/11575
Generated-by: Claude Code (Opus 5.5) ## What changes were proposed in this pull request? With many snapshots, OM startup can block for hours in `OmSnapshotLocalDataManager#init` before the RPC server starts. On a cluster with about 19k snapshots (164 GB of snapshot local data YAML, median 9 MB per file), the RPC port opened about 10 hours after a restart, with the OM main thread fully busy parsing YAML. `init` loads every YAML in file name order and, for each one, walks the previous snapshot chain loading any snapshot not yet in the version graph. A snapshot reached through that walk is loaded again when the outer loop reaches its own file. Since the file names are UUIDs, almost every YAML is parsed twice, all on one thread. This PR: * Parses every YAML exactly once at startup, keeping only its version metadata (`SnapshotVersionsMeta`), and lets `addVersionNodeWithDependents` reuse that metadata instead of reloading previous snapshots from disk. * Runs the parsing on a dedicated thread pool of `min(cores, 8)` daemon threads named `SnapshotLocalDataLoader-N`. 8 matches the `YamlSerializer` Yaml pool size, so more threads would only wait. * Leaves the handling of unreadable or mismatched YAMLs unchanged. They are still skipped, and snapshots that depend on them are still left out of the version graph. Expected improvement is about 2x from parsing once, times up to 8x from parallel parsing. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17078 ## How was this patch tested? * Extended `TestOmSnapshotLocalDataManager#testInitWithExistingYamlFiles` to count YAML loads during `init`. It fails without the fix (3 loads for 2 files) and passes with it (2 loads). * `TestOmSnapshotLocalDataManager`, `TestOmSnapshotLocalDataYaml` and `TestSnapshotDefragService` pass locally. * Checkstyle passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
