[
https://issues.apache.org/jira/browse/HDDS-17079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17079:
------------------------------
Description:
h3. Problem
To build its snapshot version graph, OmSnapshotLocalDataManager only needs each
YAML's snapshotId, previousSnapshotId, version and the previousSnapshotVersion
of each version. Every load still builds all SstFileInfo entries, which make up
almost the whole file, and then throws them away. With many snapshots this is
most of the cost of OM startup, even after HDDS-17078 (parse each YAML once at
startup).
h3. Analysis
Timing one 9 MB YAML from a cluster with about 19k snapshots (one version,
about 36.7k SstFileInfo entries), G1 with a 900 MB young generation:
||Load||1 thread, per file||8 threads, 64 files||
|Full load|0.153 s| |
|Node filter (drop the sstFiles node before construction)|0.088 s|1.6 to 3.0 s,
GC 46 to 60% of wall, old gen peak 4 GB|
|Streaming parser events|0.067 s|0.6 s, GC 1% of wall, old gen peak 9 MB|
Dropping the sstFiles node before construction is not enough: the whole node
tree is still composed first. With several loads in flight these trees get
promoted, and on a large cluster startup became GC bound.
h3. Proposed fix
Add OmSnapshotLocalDataYaml#loadWithoutSstFiles, which reads the four fields
above straight from the snakeyaml parser events and skips everything else, so
no node tree or SstFileInfo is built. The returned object has empty SST file
lists, no transaction info and no checksum. OmSnapshotLocalDataManager uses it
only in init and in the previous snapshot walk. All other reads and every write
still use the full serializer.
h3. Compatibility
The on disk format is unchanged. New code reads YAMLs written by existing
releases, and nothing new is written.
One behavior change: startup still parses every event, so files that are not
valid YAML are still skipped as before. But invalid values in other fields (for
example inside SST file entries) are no longer detected at startup; they show
up on the next full load of that snapshot.
h3. Testing
Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by
the regular serializer loads with the same snapshot ids, versions and
previousSnapshotVersion values as a full load, with empty SST file lists, and a
YAML without snapshotId is rejected.
Related to HDDS-17078; independent change, the speedups combine.
was:
h3. Problem
To build its snapshot version graph, OmSnapshotLocalDataManager only needs each
YAML's snapshotId, previousSnapshotId, version and the previousSnapshotVersion
of each version. Every load still builds all SstFileInfo entries, which make up
almost the whole file, and then throws them away. With many snapshots this is
most of the CPU cost of OM startup, even after HDDS-17078 (parse each YAML once
at startup).
h3. Analysis
Timing one 9 MB YAML from a cluster with about 19k snapshots (one version,
about 36.7k SstFileInfo entries):
* Full load: 0.153 s per file (59 MB/s)
* Load without SST file lists: 0.088 s per file (103 MB/s)
h3. Proposed fix
Add a load mode to OmSnapshotLocalDataYaml (YamlFactory(true)) that drops each
version's sstFiles node before construction, so none of its entries are built.
All other fields load as before. OmSnapshotLocalDataManager uses this mode only
in init and in the previous snapshot walk. All other reads and every write
still use the full serializer.
The serializer used in this mode refuses to save, so data loaded without SST
file lists cannot overwrite a YAML.
h3. Compatibility
The on disk format is unchanged. New code reads YAMLs written by existing
releases, and nothing new is written.
One behavior change: startup still parses the whole document, so files that are
not valid YAML are still skipped as before. But invalid values inside SST file
entries are no longer detected at startup; they show up on the next full load
of that snapshot.
h3. Testing
Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by
the regular serializer loads in the new mode with the same snapshot ids,
versions, previousSnapshotVersion values, checksum and transaction info, and
with empty SST file lists.
Related to HDDS-17078; independent change, the speedups combine.
> Skip SST file lists when loading snapshot local data YAMLs for the snapshot
> version graph
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17079
> URL: https://issues.apache.org/jira/browse/HDDS-17079
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Manager, Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> To build its snapshot version graph, OmSnapshotLocalDataManager only needs
> each YAML's snapshotId, previousSnapshotId, version and the
> previousSnapshotVersion of each version. Every load still builds all
> SstFileInfo entries, which make up almost the whole file, and then throws
> them away. With many snapshots this is most of the cost of OM startup, even
> after HDDS-17078 (parse each YAML once at startup).
> h3. Analysis
> Timing one 9 MB YAML from a cluster with about 19k snapshots (one version,
> about 36.7k SstFileInfo entries), G1 with a 900 MB young generation:
> ||Load||1 thread, per file||8 threads, 64 files||
> |Full load|0.153 s| |
> |Node filter (drop the sstFiles node before construction)|0.088 s|1.6 to 3.0
> s, GC 46 to 60% of wall, old gen peak 4 GB|
> |Streaming parser events|0.067 s|0.6 s, GC 1% of wall, old gen peak 9 MB|
> Dropping the sstFiles node before construction is not enough: the whole node
> tree is still composed first. With several loads in flight these trees get
> promoted, and on a large cluster startup became GC bound.
> h3. Proposed fix
> Add OmSnapshotLocalDataYaml#loadWithoutSstFiles, which reads the four fields
> above straight from the snakeyaml parser events and skips everything else, so
> no node tree or SstFileInfo is built. The returned object has empty SST file
> lists, no transaction info and no checksum. OmSnapshotLocalDataManager uses
> it only in init and in the previous snapshot walk. All other reads and every
> write still use the full serializer.
> h3. Compatibility
> The on disk format is unchanged. New code reads YAMLs written by existing
> releases, and nothing new is written.
> One behavior change: startup still parses every event, so files that are not
> valid YAML are still skipped as before. But invalid values in other fields
> (for example inside SST file entries) are no longer detected at startup; they
> show up on the next full load of that snapshot.
> h3. Testing
> Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by
> the regular serializer loads with the same snapshot ids, versions and
> previousSnapshotVersion values as a full load, with empty SST file lists, and
> a YAML without snapshotId is rejected.
> Related to HDDS-17078; independent change, the speedups combine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
