[
https://issues.apache.org/jira/browse/HDDS-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17912146#comment-17912146
]
Ethan Rose commented on HDDS-11745:
-----------------------------------
Based on some more discussion:
* The follower will inform the leader of filesystem snapshots that it already
has, and we will not include those in the tar
* Background filesystem snapshot services that do not go through Ratis like
sst file pruning will be stopped while the ratis snapshot is being taken on the
leader. This is more of a precaution since they are not expected to corrupt the
ratis snapshot state if the ratis snapshot is captured partway through.
* The leader should only have one ratis snapshot existing a time that it will
use to catch up any followers that need it.
** This does not introduce new space requirements. Followers need extra space
to install a ratis snapshot, so leader will have that same space already
available to create the ratis snapshot.
> Use single tar to decouple filesystem snapshot layout from OM snapshot
> transfer
> -------------------------------------------------------------------------------
>
> Key: HDDS-11745
> URL: https://issues.apache.org/jira/browse/HDDS-11745
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ethan Rose
> Assignee: Hemant Kumar
> Priority: Major
>
> The current OM snapshot creation and transfer logic uses a list of SST files
> to track hardlink structure at the source and recreate it at the destination.
> It is also used to incrementally send the snapshots to followers by only
> sending a few SST files at a time. This adds extra complexity to the OM
> snapshot flow and prevents us from easily transferring and unpacking OM DB
> snapshots in other contexts like Recon sync or debug/repair.
> Tar preserves hardlinks on pack and unpack by default, so we can get the same
> space saving by packing all required files into a single tar, and then
> unpacking at the destination. The tar can be broken into pieces and streamed
> to followers to handle cases where it is large. This decouples network
> transfer from storage format.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
