Hello Sebastien, with btrfs raid1 you get two copies of each extent on separate drives. That means you can lose one drive only, no matter how many drives are in the set. It's not traditional raid1, which is probably what you are confusing it with. Having raid1 with more than 2n-redudancy is not currently possible with btrfs, though rumour has it this capability may land in 3.8
Regards, Bart On Mon, Dec 17, 2012 at 4:51 PM, Sebastien Luttringer <sebastien.luttrin...@smartjog.com> wrote: > Hello, > > I'm testing Btrfs RAID1 feature on 3 disks of ~10GB. Last one is not > exactly 10GB (would be too easy). > > About the test machine, it's a kvm vm running an up-to-date archlinux > with linux 3.7 and btrfs-progs 0.19.20121005. > > #uname -a > Linux seblu-btrfs-1 3.7.0-1-ARCH #1 SMP PREEMPT Tue Dec 11 15:05:50 CET > 2012 x86_64 GNU/Linux > > Filesystem was created with : > > # mkfs.btrfs -L test -d raid1 -m raid1 /dev/vd[bcd] > > I downloaded a lot of linux kernel tarball and untared in into this > filesystem until it tell me enough: > > drwxr-xr-x 1 root root 330 2007-10-09 20:31 linux-2.6.23 > -rw-r--r-- 1 root root 44M 2007-10-09 20:48 linux-2.6.23.tar.bz2 > drwxr-xr-x 1 root root 344 2008-01-24 22:58 linux-2.6.24 > -rw-r--r-- 1 root root 45M 2008-01-24 23:16 linux-2.6.24.tar.bz2 > drwxr-xr-x 1 root root 352 2008-04-17 02:49 linux-2.6.25 > > Some output of btrfs tools > > # btrfs fi sh > Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 > Total devices 3 FS bytes used 11.57GB > devid 3 size 10.00GB used 9.76GB path /dev/vdd > devid 2 size 10.00GB used 10.00GB path /dev/vdc > devid 1 size 9.77GB used 9.77GB path /dev/vdb > > # btrfs fi df . > Data, RAID1: total=13.50GB, used=10.56GB > Data: total=8.00MB, used=0.00 > System, RAID1: total=8.00MB, used=4.00KB > System: total=4.00MB, used=0.00 > Metadata, RAID1: total=1.25GB, used=1.01GB > Metadata: total=8.00MB, used=0.00 > > I have choosen a _raid1_ filesystem, so I expect every disk be able to > die and fs still usuable without data lost. > > I used chksfv[1] to make checksum of tarball on disk and check my data > are safe. > > I killed the first disk via libvirt. > # virsh detach-disk seblu-btrfs-1 vdb > > Btrfs detect missing disk \o/ > > # btrfs fi sh > Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 > Total devices 3 FS bytes used 11.57GB > devid 3 size 10.00GB used 8.01GB path /dev/vdd > devid 2 size 10.00GB used 9.26GB path /dev/vdc > *** Some devices missing > > Listing all files and cksum on tarballs are good. > > # cksfv -f ~/checksum > Everything OK > > We have raid1, so we can expect kill one disk (the smaller). Let's go. > > # virsh detach-disk seblu-btrfs-1 vdd > > # btrfs fi sh > Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 > Total devices 3 FS bytes used 11.57GB > devid 3 size 10.00GB used 8.01GB path /dev/vdd > devid 2 size 10.00GB used 9.26GB path /dev/vdc > *** Some devices missing > > It doesn't see the drive go away... Listing doesn't works. > > # find /mnt/test > . > ./pkg > ^C > # cksfv -f ~/checksum > linux-2.6.30.tar.bz2 OK > linux-2.6.31.tar.bz2 Input/output error > linux-2.6.32.tar.bz2 Input/output error > linux-2.6.33.tar.bz2 Input/output error > > So data and metada seems broken. Filesystem is become read only > # touch séb > touch: cannot touch ‘séb’: Read-only file system > > and finally I got a traceback in dmesg > > [ 804.910405] ------------[ cut here ]------------ > [ 804.910491] WARNING: at fs/btrfs/super.c:246 > __btrfs_abort_transaction+0xdf/0x100 [btrfs]() > [ 804.910499] Hardware name: Bochs > [ 804.910501] btrfs: Transaction aborted > [ 804.910502] Modules linked in: cirrus ttm i2c_piix4 drm_kms_helper > psmouse intel_agp drm evdev intel_gtt serio_raw syscopyarea sysfillrect > sysimgblt i2c_core processor button pcspkr microcode btrfs crc32c > libcrc32c zlib_deflate ata_generic pata_acpi virtio_balloon virtio_blk > virtio_net ata_piix libata uhci_hcd virtio_pci virtio_ring virtio > usbcore usb_common scsi_mod floppy > [ 804.910575] Pid: 2077, comm: touch Not tainted 3.7.0-1-ARCH #1 > [ 804.910577] Call Trace: > [ 804.910606] [<ffffffff8105742f>] warn_slowpath_common+0x7f/0xc0 > [ 804.910610] [<ffffffff81057526>] warn_slowpath_fmt+0x46/0x50 > [ 804.910619] [<ffffffffa015ff6f>] > __btrfs_abort_transaction+0xdf/0x100 [btrfs] > [ 804.910635] [<ffffffffa01802c0>] ? verify_parent_transid+0x170/0x170 > [btrfs] > [ 804.910648] [<ffffffffa016402a>] __btrfs_cow_block+0x48a/0x510 [btrfs] > [ 804.910661] [<ffffffffa0164227>] btrfs_cow_block+0xf7/0x230 [btrfs] > [ 804.910672] [<ffffffffa016752b>] push_leaf_right+0x11b/0x190 [btrfs] > [ 804.910681] [<ffffffffa0167cb1>] split_leaf+0x621/0x740 [btrfs] > [ 804.910690] [<ffffffffa0160d52>] ? leaf_space_used+0xd2/0x110 [btrfs] > [ 804.910710] [<ffffffffa01b9455>] ? > btrfs_set_lock_blocking_rw+0xb5/0x120 [btrfs] > [ 804.910720] [<ffffffffa016894c>] btrfs_search_slot+0x89c/0x900 [btrfs] > [ 804.910730] [<ffffffffa0169fcc>] btrfs_insert_empty_items+0x7c/0xe0 > [btrfs] > [ 804.910742] [<ffffffffa017c313>] insert_with_overflow+0x43/0x120 [btrfs] > [ 804.910753] [<ffffffffa017c4ac>] btrfs_insert_dir_item+0xbc/0x200 > [btrfs] > [ 804.910767] [<ffffffffa0195c6b>] btrfs_add_link+0xeb/0x300 [btrfs] > [ 804.910781] [<ffffffffa0196f64>] btrfs_create+0x1a4/0x210 [btrfs] > [ 804.910799] [<ffffffff81217a0c>] ? security_inode_permission+0x1c/0x30 > [ 804.910813] [<ffffffff8118fd56>] vfs_create+0xb6/0x120 > [ 804.910833] [<ffffffff8119281d>] do_last+0xd5d/0xeb0 > [ 804.910838] [<ffffffff8118ed68>] ? link_path_walk+0x78/0x950 > [ 804.910841] [<ffffffff81192a2c>] path_openat+0xbc/0x530 > [ 804.910855] [<ffffffff8114b551>] ? handle_mm_fault+0x261/0x330 > [ 804.910871] [<ffffffff814a6314>] ? __do_page_fault+0x2d4/0x590 > [ 804.910875] [<ffffffff81193211>] do_filp_open+0x41/0xa0 > [ 804.910878] [<ffffffff811a0119>] ? __alloc_fd+0xe9/0x140 > [ 804.910887] [<ffffffff81182fc4>] do_sys_open+0xf4/0x1e0 > [ 804.910890] [<ffffffff814a65de>] ? do_page_fault+0xe/0x10 > [ 804.910895] [<ffffffff811830d1>] sys_open+0x21/0x30 > [ 804.910899] [<ffffffff814aa51d>] system_call_fastpath+0x1a/0x1f > [ 804.910904] ---[ end trace b713d23404f7cc1c ]--- > [ 804.910911] BTRFS error (device vdc) in __btrfs_cow_block:1014: IO > failure > [ 804.912156] btrfs is forced readonly > > I get the feeling that RAID1 only allow one disk removing. Which is more > a RAID5 feature. > I'm afraid Btrfs raid1 will not be working before the end of the world. > Sad day. > > Cheers, > > [1] https://www.archlinux.org/packages/community/i686/cksfv/ > > -- > Sebastien "Seblu" Luttringer > Research & Engineering Manager - Smartjog - TDF > 27 bd Hippolyte Marques, 94200 Ivry sur Seine, France > Phone: +33.1.58.68.62.32 / Fax: +33.1.58.68.62.05 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
