Siyao Meng created HDDS-17079:
---------------------------------
Summary: Skip SST file lists when loading snapshot local data
YAMLs for the snapshot version graph
Key: HDDS-17079
URL: https://issues.apache.org/jira/browse/HDDS-17079
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Manager, Snapshot
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Problem
To build its snapshot version graph, OmSnapshotLocalDataManager only needs each
YAML's snapshotId, previousSnapshotId, version and the previousSnapshotVersion
of each version. Every load still builds all SstFileInfo entries, which make up
almost the whole file, and then throws them away. With many snapshots this is
most of the CPU cost of OM startup, even after HDDS-17078 (parse each YAML once
at startup).
h3. Analysis
Timing one 9 MB YAML from a cluster with about 19k snapshots (one version,
about 36.7k SstFileInfo entries):
* Full load: 0.153 s per file (59 MB/s)
* Load without SST file lists: 0.088 s per file (103 MB/s)
h3. Proposed fix
Add a load mode to OmSnapshotLocalDataYaml (YamlFactory(true)) that drops each
version's sstFiles node before construction, so none of its entries are built.
All other fields load as before. OmSnapshotLocalDataManager uses this mode only
in init and in the previous snapshot walk. All other reads and every write
still use the full serializer.
The serializer used in this mode refuses to save, so data loaded without SST
file lists cannot overwrite a YAML.
h3. Compatibility
The on disk format is unchanged. New code reads YAMLs written by existing
releases, and nothing new is written.
One behavior change: startup still parses the whole document, so files that are
not valid YAML are still skipped as before. But invalid values inside SST file
entries are no longer detected at startup; they show up on the next full load
of that snapshot.
h3. Testing
Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by
the regular serializer loads in the new mode with the same snapshot ids,
versions, previousSnapshotVersion values, checksum and transaction info, and
with empty SST file lists.
Depends on HDDS-17078.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
