[
https://issues.apache.org/jira/browse/HDDS-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17078:
----------------------------------
Labels: pull-request-available (was: )
> OM startup takes hours with many snapshots because snapshot local data YAMLs
> are parsed repeatedly on one thread
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17078
> URL: https://issues.apache.org/jira/browse/HDDS-17078
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Manager, Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> On a cluster with about 19k snapshots, OM takes about 10 hours to open its
> RPC port after a restart. During that time the OM main thread is fully busy
> (100% of one core) in OmSnapshotLocalDataManager initialization:
> {code}
> HashMap.put <- SafeConstructor.processDuplicateKeys / flattenMapping
> OmSnapshotLocalDataYaml$SnapshotLocalDataConstructor$ConstructSstFileInfo.construct
> <- ConstructVersionMeta.construct <- ConstructSnapshotLocalData.construct
> YamlSerializer.load
> OmSnapshotLocalDataManager.tryLoadSnapshotLocalData
> OmSnapshotLocalDataManager.addVersionNodeWithDependents
> OmSnapshotLocalDataManager.init
> OmSnapshotLocalDataManager.<init>
> OmSnapshotManager.<init>
> OzoneManager.instantiateServices
> {code}
> The whole delay falls between "Ozone filesystem snapshot feature is enabled"
> and the next OM startup log line. The rest of startup takes seconds. This
> affects 2.2.0 and current master.
> h3. Analysis
> OmSnapshotLocalDataManager#init lists all snapshot local data YAMLs, sorts
> them by file name and loads each one. For each file,
> addVersionNodeWithDependents walks the previous snapshot chain and loads
> every snapshot YAML not yet in the version graph. A snapshot reached through
> that walk is loaded again when the outer loop reaches its own file. Since the
> file names are UUIDs, the order is effectively random with respect to the
> chain, so almost every YAML is parsed twice, all on one thread.
> Measurements on the affected cluster:
> * 19,358 YAMLs, 164 GB in total (median 9 MB, max 17.7 MB). A median file
> holds one version with about 36.7k SstFileInfo entries.
> * Observed parse rate during startup: about 10 MB/s, 1.4 s per file.
> * About 2 x 164 GB at 10 MB/s comes to about 9 hours, which matches the
> observed startup time.
> h3. Proposed fix
> Parse every YAML exactly once in init, keeping only its version metadata
> (SnapshotVersionsMeta), and let the dependency walk reuse that metadata
> instead of reloading previous snapshots from disk. Run the parsing on a
> dedicated thread pool of min(cores, 8) threads; 8 matches the YamlSerializer
> Yaml pool size, so more threads would only wait. Handling of unreadable or
> mismatched YAMLs is unchanged.
> Expected improvement: about 2x from parsing once, times up to 8x from
> parallel parsing.
> h3. Testing
> Extended TestOmSnapshotLocalDataManager#testInitWithExistingYamlFiles to
> count YAML loads during init. It fails on the current code (3 loads for 2
> files) and passes with the fix (2 loads).
> h3. Related
> The YAMLs themselves are much larger than expected because each snapshot
> lists tens of thousands of small SST files. That will be tracked separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
