smengcl opened a new pull request, #10679:
URL: https://github.com/apache/ozone/pull/10679
## What changes were proposed in this pull request?
A bad local YAML should not make OM startup fail. For snapshots with many
small SST files, this YAML can grow beyond SnakeYAML's default 3 MiB code-point
limit. When that happens, loading the YAML can fail during OM startup or
snapshotLocalData initialization.
- Raise the `OmSnapshotLocalDataYaml` parser limit to 128 MB (from snakeyaml
default of 3 MB).
- Each SstFileInfo represents one SST file. For FSO, it takes roughly 256
bytes to store each SstFileInfo in yaml with the current schema. So 3 MB had
room for 11718 SST files, and 128 MB would have room for 500000 SST files in
one snapshot (RocksDB checkpoint).
- During OM startup, skip unloadable or mismatched snapshot local data YAMLs
instead of failing initialization.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15760
## How was this patch tested?
- Add tests for large yaml loading and invalid yaml files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
