GeorgeJahad opened a new pull request, #5014: URL: https://github.com/apache/ozone/pull/5014
## What changes were proposed in this pull request? The incremental checkpointing PR I created here: https://github.com/apache/ozone/pull/4770 allows multiple increments of data if the initial tarball took too long to download. This is needed for om snapshots but is not completely sufficient, because the first increment can grow quite large, as it includes all sst files that exist at the time it is created. (Each subsequent increment only includes the sst files created after the first tarball was created.) This means there is no limit to the size of the initial tarball. In order to alleviate that, this PR adds the "max sst size" config flag: **OZONE_OM_RATIS_SNAPSHOT_MAX_TOTAL_SST_SIZE_KEY**. The tarball creation process has been modified to close the tarball once it reaches that size of sst files. If that leaves the tarball incomplete, the follower will know to request another, incremental one. Note that with this design all the non sst files are in the final tarball; (because they are mutable and we want the latest version of each). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-8943 ## How was this patch tested? Tests were updated accordingly. However due to another bug, one of the test classes is entirely disabled: https://issues.apache.org/jira/browse/HDDS-8952 It will be reenabled once that ticket is addressed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
