Hi Venky, Thank's a lot for these explanations. I had some trouble when upgrading to v16.2.5. I'm using debian 10 with cephadm and the 16.2.5 containers use generated a lot of network dropped packets (I don't know why) on all my OSD hosts. I encountered also some hangs while reading files in cephfs (may be related with the dropped packets), so I downgraded our 2 clusters to 16.2.4. I will try to resolve my problems with 16.2.5 to benefit from the next enhancements of snapshots mirroring...
Kind regards, Arnaud ----- Mail original ----- De: "Venky Shankar" <[email protected]> À: "arnaud martel" <[email protected]> Cc: "ceph-users" <[email protected]> Envoyé: Lundi 23 Août 2021 14:39:15 Objet: Re: [ceph-users] cephfs snapshots mirroring On Mon, Aug 23, 2021 at 5:36 PM Arnaud MARTEL <[email protected]> wrote: > > Hi all, > > I'm not sure to really understand how cephfs snapshots mirroring is supposing > to work. > > I have 2 ceph clusters (pacific 16.2.4) and snapshots mirroring is set up for > only one directory, /ec42/test, in our cephfs filesytem (it's for test > purposes but we plan to use it with about 50-60 directories and 1.5 PB). > I have also set up a pool with erasure coding and configured the layout for > my /ec42 directory to use the EC pool (on both clusters). > > I used the following steps to test the snapshot mirroring: > - copy about 70GB in /ec42/test on source cluster > - create a snapshot (mkdir /ec42/test/.snap/snap1) > - remove 5 text files from /ec42/test (the total files size is about 5-10 KB) > - create another snapshot ( mkdir /ec42/test/.snap/snap2) > > What I see during cephfs-mirror execution: > - after snap1 creation, 70 GB are transferred to to target cluster, then > snapshot (snap1) is created on target cluster > - after snap2 creation, the remote directory (on target cluster) is emptied, > then 70GB are transferred again and, finally, the second snapshot (snap2) is > created > > I thought that only the diff between the snapshots would be transferred (or > remove) but it seems that all data in the source snapshot are pushed each > time. Is it the design of the snapshot mirroring feature or have I missed > something? The lack of an efficient snapshot diff feature in CephFS requires the mirror daemon to walk the entire snapshot. That said, with 16.2.5 release, cephfs-mirror only transfers those files that have been modified between two snapshots (16.2.4 does a remote data purge, followed by a full copy), however, to figure out what files have changed it still needs to walk the entire directory tree. Efficient snapshot diff for CephFS is been worked on with priority. > I wanted to use snapshot mirroring to backup our cephfs filesystem but it > will be impossible if we have to transfert 1,5PB every day. > And any other suggestion to backup cephfs with 1,5PB would be very helpful... > > Arnaud > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
