Re: btrfs send/receive vs rsync
Marc MERLIN posted on Fri, 29 Jun 2018 09:24:20 -0700 as excerpted: >> If instead of using a single BTRFS filesystem you used LVM volumes >> (maybe with Thin provisioning and monitoring of the volume group free >> space) for each of your servers to backup with one BTRFS filesystem per >> volume you would have less snapshots per filesystem and isolate >> problems in case of corruption. If you eventually decide to start from >> scratch again this might help a lot in your case. > > So, I already have problems due to too many block layers: > - raid 5 + ssd - bcache - dmcrypt - btrfs > > I get occasional deadlocks due to upper layers sending more data to the > lower layer (bcache) than it can process. I'm a bit warry of adding yet > another layer (LVM), but you're otherwise correct than keeping smaller > btrfs filesystems would help with performance and containing possible > damage. > > Has anyone actually done this? :) So I definitely use (and advocate!) the split-em-up strategy, and I use btrfs, but that's pretty much all the similarity we have. I'm all ssd, having left spinning rust behind. My strategy avoids unnecessary layers like lvm (tho crypt can arguably be necessary), preferring direct on-device (gpt) partitioning for simplicity of management and disaster recovery. And my backup and recovery strategy is an equally simple mkfs and full-filesystem-fileset copy to an identically sized filesystem, with backups easily bootable/mountable in place of the working copy if necessary, and multiple backups so if disaster takes out the backup I was writing at the same time as the working copy, I still have a backup to fall back to. So it's different enough I'm not sure how much my experience will help you. But I /can/ say the subdivision is nice, as it means I can keep my root filesystem read-only by default for reliability, my most-at-risk log filesystem tiny for near-instant scrub/balance/check, and my also at risk home small as well, with the big media files being on a different filesystem that's mostly read-only, so less at risk and needing less frequent backups. The tiny boot and large updates (distro repo, sources, ccache) are also separate, and mounted only for boot maintenance or updates. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send/receive vs rsync
On Fri, Jun 29, 2018 at 10:04:02AM +0200, Lionel Bouton wrote: > Hi, > > On 29/06/2018 09:22, Marc MERLIN wrote: > > On Fri, Jun 29, 2018 at 12:09:54PM +0500, Roman Mamedov wrote: > >> On Thu, 28 Jun 2018 23:59:03 -0700 > >> Marc MERLIN wrote: > >> > >>> I don't waste a week recreating the many btrfs send/receive relationships. > >> Consider not using send/receive, and switching to regular rsync instead. > >> Send/receive is very limiting and cumbersome, including because of what you > >> described. And it doesn't gain you much over an incremental rsync. As for > > Err, sorry but I cannot agree with you here, at all :) > > > > btrfs send/receive is pretty much the only reason I use btrfs. > > rsync takes hours on big filesystems scanning every single inode on both > > sides and then seeing what changed, and only then sends the differences > > It's super inefficient. > > btrfs send knows in seconds what needs to be sent, and works on it right > > away. > > I've not yet tried send/receive but I feel the pain of rsyncing millions > of files (I had to use lsyncd to limit the problem to the time the > origin servers reboot which is a relatively rare event) so this thread > picked my attention. Looking at the whole thread I wonder if you could > get a more manageable solution by splitting the filesystem. So, let's be clear. I did backups with rsync for 10+ years. It was slow and painful. On my laptop an hourly rsync between 2 drives slowed down my machine to a crawl while everything was being stat'ed, it took forever. Now with btrfs send/receive, it just works, I don't even see it happening in the background. Here is a page I wrote about it in 2014: http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive Here is a talk I gave in 2014 too, scroll to the bottom of the page, and the bottom of the talk outline: http://marc.merlins.org/perso/btrfs/2014-05.html#My-Btrfs-Talk-at-Linuxcon-JP-2014 and click on 'Btrfs send/receive' > If instead of using a single BTRFS filesystem you used LVM volumes > (maybe with Thin provisioning and monitoring of the volume group free > space) for each of your servers to backup with one BTRFS filesystem per > volume you would have less snapshots per filesystem and isolate problems > in case of corruption. If you eventually decide to start from scratch > again this might help a lot in your case. So, I already have problems due to too many block layers: - raid 5 + ssd - bcache - dmcrypt - btrfs I get occasional deadlocks due to upper layers sending more data to the lower layer (bcache) than it can process. I'm a bit warry of adding yet another layer (LVM), but you're otherwise correct than keeping smaller btrfs filesystems would help with performance and containing possible damage. Has anyone actually done this? :) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-send-receive vs rsync for (incremental/full) backups
On Fri, Dec 13, 2013 at 02:30:26PM -0700, Chris Murphy wrote: > > On Dec 13, 2013, at 1:35 PM, Hugo Mills wrote: > > > On Fri, Dec 13, 2013 at 07:44:16PM +, Martin wrote: > >> OK... So for backing up across a local network to a second physical host... > >> > >> Is btrfs-send-receive stable enough now to be used? > > > > It seems to be OK for me at the moment -- I'm not running it across > > a network (yet), but I am using it between different filesystems on > > the same host, using pipes. > Question. Are the source file system's checksums (and metadata) > preserved in the send file? I don't think checksums are transported (from what I remember of the FAR format). > Or does the send file just contain data, and checksums are > recomputed on receive? I'm curious if the send file can be stored on > a non-checksumming file system, and yet have a means to verify the > integrity of the files once received. Either the checksums in the > send file being check during receive, or subsequent to receive > completion by using scrub? The FAR format contains a sequence of what amount to FS commands: there are equivalents of cp, rm, chmod, chown, setfattr, cat (of data inline in the FAR file). There's also a data type that clones an extent from a file in another subvolume with a given UUID. Assuming you have some way of keeping track of subvolume UUIDs on an FS that doesn't support them, there's no reason you can't generate or unpack FAR files on a non-checksumming (and non-CoW) FS. > And yes I can confirm that send outputting to a file on a 2nd drive, > and receiving it to yet another drive does work, and the source and > destination subvolumes are the same in seemingly every way. I didn't > separately rsync checksum compare the two subvolumes, however. That > seems like possibly a good test. I also haven't tried doing it over > a network. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Once is happenstance; twice is coincidence; three times --- is enemy action. signature.asc Description: Digital signature
Re: btrfs-send-receive vs rsync for (incremental/full) backups
On Dec 13, 2013, at 1:35 PM, Hugo Mills wrote: > On Fri, Dec 13, 2013 at 07:44:16PM +, Martin wrote: >> OK... So for backing up across a local network to a second physical host... >> >> Is btrfs-send-receive stable enough now to be used? > > It seems to be OK for me at the moment -- I'm not running it across > a network (yet), but I am using it between different filesystems on > the same host, using pipes. Question. Are the source file system's checksums (and metadata) preserved in the send file? Or does the send file just contain data, and checksums are recomputed on receive? I'm curious if the send file can be stored on a non-checksumming file system, and yet have a means to verify the integrity of the files once received. Either the checksums in the send file being check during receive, or subsequent to receive completion by using scrub? And yes I can confirm that send outputting to a file on a 2nd drive, and receiving it to yet another drive does work, and the source and destination subvolumes are the same in seemingly every way. I didn't separately rsync checksum compare the two subvolumes, however. That seems like possibly a good test. I also haven't tried doing it over a network. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-send-receive vs rsync for (incremental/full) backups
On Fri, Dec 13, 2013 at 07:44:16PM +, Martin wrote: > OK... So for backing up across a local network to a second physical host... > > Is btrfs-send-receive stable enough now to be used? It seems to be OK for me at the moment -- I'm not running it across a network (yet), but I am using it between different filesystems on the same host, using pipes. > How does send-receive compare to using rsync for backups? It's more fiddly to get right, and needs a bit more thought about how to set it up sanely. I have a python library that I use to write my backup scripts. I'll have to write it up one of these days... > Any comments please from those using such things? Works for me, so far, as far as I know. Although we've got recent reports of *something* still going wrong with systemd journal files. No investigation of that yet, though. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- In one respect at least, the Martians are a happy people: --- they have no lawyers. signature.asc Description: Digital signature
btrfs-send-receive vs rsync for (incremental/full) backups
OK... So for backing up across a local network to a second physical host... Is btrfs-send-receive stable enough now to be used? How does send-receive compare to using rsync for backups? Any comments please from those using such things? Thanks, Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
