Re: Copy/move btrfs volume
On 1 July 2010 11:28, Lubos Kolouch lubos.kolo...@gmail.com wrote: Hello, I am testing btrfs on one of our backup servers (many millions of files, 1.5TB size, running on (non-btrfs-provided-) raid5). I am using subvolumes/snapshots with following rsync. It works very well, but I would like to ask a question... say I would need to copy/move the files to different server/disk. Normally I would do it with rsync, but I guess it will not preserve the subvolumes, it will also not detect that they are the same files (I guess they are not just normal hardlinks). So I would end up with duplicated files. What is the correct way to do this? The only way to do this preserving duplication is to use hardlinks between duplicated files (which reference counts the inode), and use 'rsync -H'. Dan -- Daniel J Blueman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copy/move btrfs volume
Daniel J Blueman, Thu, 01 Jul 2010 12:26:10 +0100: What is the correct way to do this? The only way to do this preserving duplication is to use hardlinks between duplicated files (which reference counts the inode), and use 'rsync -H'. Dan But when the files are on different snaphots, does rsync see them as hardlinked? A scenario - I have raid5 of say, 1TB HDDs. It contains many snapshots. Then, few years later, new machine is bought and there are, say, 5TB discs. So I need to transfer the btrfs volume to the new machine. But how to do it so that it looks the *same*, ie. the same snapshots? I could of course write a custom script to create the subvolume, rsync the files, create snapshot, rsync files, etc, but it would be nice if the btrfs toolset supports this by default... Lubos -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: volume broken? btrfsck fails
Yee-Ting Li yee379 at gmail.com writes: Hi, i think my btrfs volume is hosed it mounts okay, but iostat shows /dev/sdg on 100% load. dmesg shows lots of 'parent transid verify failed on x wanted y found z'. then after a while i can't read from it (access to the filesystem freezes). the machine had crashed (prob from some other process), and upon reboot i've been experience this problem since. can anyone provide any guidance in how to proceed? cheers, Yee. I am also having the same problem with a slightly different setup. In My case I cannot mount the filesystem. mount, btrfs-endio-met and kblockd/0 will all continually run until the system freezes up and requires a power cycle. I have both the kernel module and the tools checked out from git so if you have any ideas on fix's I can build them and test it out. here is some information about my setup [r...@solution ~]# uname -a Linux solution.bcig 2.6.35-0.13.rc3.git2.fc14.x86_64 #1 SMP Mon Jun 28 19:27:35 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux [r...@solution ~]# [r...@solution ~]# btrfs-show Label: store uuid: 4ba1cc6b-e12a-454a-a064-f4019312c063 Total devices 7 FS bytes used 1.15TB devid1 size 931.51GB used 415.55GB path /dev/sdb devid2 size 931.51GB used 518.50GB path /dev/sdc devid3 size 931.51GB used 342.04GB path /dev/sdd devid4 size 931.51GB used 523.54GB path /dev/sde devid5 size 465.76GB used 402.54GB path /dev/sdf devid6 size 465.76GB used 382.54GB path /dev/sdg devid7 size 465.76GB used 367.54GB path /dev/sdh Btrfs v0.19-16-g075587c-dirty [r...@solution ~]# [r...@solution ~]# tail -n 12 /var/log/messages Jul 1 04:47:03 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: verify_parent_transid: 9244 callbacks suppressed Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 Jul 1 04:47:08 solution kernel: parent transid verify failed on 1682196926464 wanted 285263 found 283510 [r...@solution ~]# -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copy/move btrfs volume
On 07/01/2010 05:33 AM, Lubos Kolouch wrote: Daniel J Blueman, Thu, 01 Jul 2010 12:26:10 +0100: What is the correct way to do this? The only way to do this preserving duplication is to use hardlinks between duplicated files (which reference counts the inode), and use 'rsync -H'. Dan Hello, With backed up files consisting of hard links, I usually use dd to copy the file systems at the block level # dd if=/dev/sda of=/dev/sdb bs=20M and then expand the file system. This is because I found that tools like rsync, while usually fast, are extremely slow when dealing with millions of hard linked files. This could also be used for btrfs to keep its snapshots. A scenario - I have raid5 of say, 1TB HDDs. It contains many snapshots. Then, few years later, new machine is bought and there are, say, 5TB discs. ... Lubos For me, I had to copy over BackupPC hardlinked files from a full disk to a smaller disk, both using ext4, and I could not use dd. What normally should have taken an hour, instead took almost a week. (Yes, I wanted to use btrfs, but it had a hard link limit of 255 - don't know if it still does.) It would be nice to have a btrfs command that could rapidly copy over the file system, snapshots, and all other file system info. But what benefit would having a native btrfs 'copy/rsync' command have over the dd/resize option? Pros - Files will be immediately checksumed on new disks, but this may not be as important since a checksum/verify command will be implemented. - Great 'feature' for copying files to new drives, and keeping snapshots. Could even be used to export snapshots. - I believe compressed files will have to be uncompressed and recompressed, depending on when file is checksummed. (I may be wrong on this one). This will actually be a con for slow and/or high load machines. - One command instead of many (dd - resize - verify). Cons - File system would still have to be unmounted, or at least read-only, as I doubt the command will have rsync's update or delete abilities. But, maybe it could. Questionable - May be faster than dd/resize, or it may be just as slow as rsync is with hard links. And I am talking about dozens to thousands of snapshots, and millions to billions of files. Matt -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copy/move btrfs volume
On Thu, Jul 01, 2010 at 11:33:59AM +, Lubos Kolouch wrote: Daniel J Blueman, Thu, 01 Jul 2010 12:26:10 +0100: What is the correct way to do this? The only way to do this preserving duplication is to use hardlinks between duplicated files (which reference counts the inode), and use 'rsync -H'. Dan But when the files are on different snaphots, does rsync see them as hardlinked? A scenario - I have raid5 of say, 1TB HDDs. It contains many snapshots. Then, few years later, new machine is bought and there are, say, 5TB discs. So I need to transfer the btrfs volume to the new machine. But how to do it so that it looks the *same*, ie. the same snapshots? I could of course write a custom script to create the subvolume, rsync the files, create snapshot, rsync files, etc, but it would be nice if the btrfs toolset supports this by default... This is definitely something I'm looking to add. The btrfs-progs git tree has some code that allows userland to walk the btrees and detect the duplicate files. But this is just a building block needed for the full backup program. Instead of hard links, it is possible to use reflinks with cp, which uses the cloning ioctl. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Atomic replacement of subvolumes is not possible
On Wed, Jun 30, 2010 at 09:26:11AM -0500, C Anthony Risinger wrote: On Wed, Jun 30, 2010 at 8:31 AM, Chris Mason chris.ma...@oracle.com wrote: On Sun, Jun 27, 2010 at 07:44:12PM -0500, C Anthony Risinger wrote: On Sat, Jun 26, 2010 at 12:25 PM, Daniel Baumann dan...@debian.org wrote: Hi, this is basically a forward from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587253 rename(2) allows for the atomic replacement of files. Being able to atomically replace subvolume snapshots would be equally invaluable, since it would permit lock-free replacement of subvolumes. % btrfs subvolume snapshot src dest creates dest as a snapshot of src. However, if I want to do the converse, % btrfs subvolume snapshot dest src then dest is snapshotted as src/dest, i.e. not replacing the original subvolume, but going inside the original subvolume. Use case 1: I have a subvolume of data under active use, which I want to periodically update. I'd like to do this by atomically replacing its contents. I can replace the content right now by deleting the old subvolume and then snapshotting the new on in its place, but it's racy. It really needs to be replaced in a single operation, or else there's a small window where there is no data, and I'd need to resort to some external locking to protect myself. I'm not sure I understand use case #1. The problem is that you'll have files open in the subvolume and you can't just pull the rug out from under them. Could you tell me a little more about what you're trying to do? Use case 2: In schroot, we create btrfs subvolume snapshots to get copy-on- write chroots. This works just fine. We also provide direct access to the source subvolume, but since it could be snapshotted in an inconsistent state while being updated, we want to do the following: · snapshot source subvolume · update snapshot · replace source volume with updated snapshot Please keep roger in the cc for any replies, thanks. i am also looking for functionality similar to this, except i would like to be able to replace the DEFAULT subvolume, with an empty or existing subvolume, and put the original default subvolume INSIDE the new root (or drop it completely), outlined by this post and the thread it's in: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg05278.html is there any feedback on these actions? no one seems to even respond :-( it would seem we need ways to swap subvolumes around, _including_ the default, providing the on-disk format supports such operations. Moving 'default' generally involves a reboot for the same reasons. We have to worry about open files and their view of the filesystem. mv on a directory won't affect file handles that are open, and renaming subvolumes needs to follow a similar model. could we fail if the user tries to replace a subvolume while it's being used? what if the root device is _not_ the default (.) subvolume, then can it be swapped? in my use case, i am running in initramfs, so the root device has not even been mounted or pivoted to; it should be safe to do whatever i want to the filesystem. i want to move the user's installation to a dedicated subvolume. what about this: would it be possible to have TWO subvolumes by default? the regular one (current directory, .): mount -o subvol=. btrfs_dev /mnt would behave as it does now. BUT... there would then be a special, permanent (like . is right now) subvol, say parent directory (..): mount -o subvol=.. btrfs_dev /mnt TWO dots would mount the parent of ., where i could then swap out the real default (.). would that work? We do provide a set-default ioctl that can be used to change the default for the next mount. This is pretty close to what you want, let me think about ways to make it easier to use. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
