Re: [Question] Btrfs on iSCSI device
Hi, On 06/27/2014 05:44 PM, Zhe Zhang wrote: Hi, I setup 2 Linux servers to share the same device through iSCSI. Then I created a btrfs on the device. Then I saw the problem that the 2 Linux servers do not see a consistent file system image. Details: -- Server 1 running kernel 2.6.32, server 2 running 3.2.1 -- Both running btrfs v0.20-rc1 -- Server 2 has device /dev/vdc, exposed as iSCSI target -- Server 1 mounts the device as /dev/sda -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs', -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any file under /mnt/btrfs -- I created /mnt/btrfs/foo on server 2 as well; then I added some content from both server 1 and server 2 to /mnt/btrfs/foo -- After that each server sees the content it adds, but not the content from the other server -- Both server 'umount /mnt/btrfs', and mount it again -- Then both servers see /mnt/btrfs/foo with the content added from server 2 (I guess it's because server 2 created the foo file later than server 1). I did a similar test on ext4 and both servers see a consistent image of the file system. When server 1 creates a foo file server 2 immediately sees it. Is this how btrfs is supposed to work? I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem. The fact that you see it working, I suspect that is is casual. When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time. Thanks, Zhe -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Question] Btrfs on iSCSI device
On 2014-06-27 12:34, Goffredo Baroncelli wrote: Hi, On 06/27/2014 05:44 PM, Zhe Zhang wrote: Hi, I setup 2 Linux servers to share the same device through iSCSI. Then I created a btrfs on the device. Then I saw the problem that the 2 Linux servers do not see a consistent file system image. Details: -- Server 1 running kernel 2.6.32, server 2 running 3.2.1 -- Both running btrfs v0.20-rc1 -- Server 2 has device /dev/vdc, exposed as iSCSI target -- Server 1 mounts the device as /dev/sda -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs', -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any file under /mnt/btrfs -- I created /mnt/btrfs/foo on server 2 as well; then I added some content from both server 1 and server 2 to /mnt/btrfs/foo -- After that each server sees the content it adds, but not the content from the other server -- Both server 'umount /mnt/btrfs', and mount it again -- Then both servers see /mnt/btrfs/foo with the content added from server 2 (I guess it's because server 2 created the foo file later than server 1). I did a similar test on ext4 and both servers see a consistent image of the file system. When server 1 creates a foo file server 2 immediately sees it. Is this how btrfs is supposed to work? I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem. The fact that you see it working, I suspect that is is casual. When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time. Thanks, Zhe -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html If you need shared storage like that, you need to use a real cluster filesystem like GFS2 or OCFS2, BTRFS isn't designed for any kind of concurrent access to shared storage from separate systems. The reason it appears to work when using iSCSI and not with directly connected parallel SCSI or SAS is that iSCSI doesn't provide low level hardware access. smime.p7s Description: S/MIME Cryptographic Signature
Re: [Question] Btrfs on iSCSI device
On Fri, Jun 27, 2014 at 1:15 PM, Austin S Hemmelgarn ahferro...@gmail.com wrote: On 2014-06-27 12:34, Goffredo Baroncelli wrote: Hi, On 06/27/2014 05:44 PM, Zhe Zhang wrote: Hi, I setup 2 Linux servers to share the same device through iSCSI. Then I created a btrfs on the device. Then I saw the problem that the 2 Linux servers do not see a consistent file system image. Details: -- Server 1 running kernel 2.6.32, server 2 running 3.2.1 -- Both running btrfs v0.20-rc1 -- Server 2 has device /dev/vdc, exposed as iSCSI target -- Server 1 mounts the device as /dev/sda -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs', -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any file under /mnt/btrfs -- I created /mnt/btrfs/foo on server 2 as well; then I added some content from both server 1 and server 2 to /mnt/btrfs/foo -- After that each server sees the content it adds, but not the content from the other server -- Both server 'umount /mnt/btrfs', and mount it again -- Then both servers see /mnt/btrfs/foo with the content added from server 2 (I guess it's because server 2 created the foo file later than server 1). I did a similar test on ext4 and both servers see a consistent image of the file system. When server 1 creates a foo file server 2 immediately sees it. Is this how btrfs is supposed to work? I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem. The fact that you see it working, I suspect that is is casual. When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time. Thanks, Zhe -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html If you need shared storage like that, you need to use a real cluster filesystem like GFS2 or OCFS2, BTRFS isn't designed for any kind of concurrent access to shared storage from separate systems. The reason it appears to work when using iSCSI and not with directly connected parallel SCSI or SAS is that iSCSI doesn't provide low level hardware access. I did more testing with ext4 and it supports what Goffredo and Austin said above. Error message is cannot access xxx: Input/output error. It seems to me that both servers hold some file system data structures in memory and eventually conflict with each other (like writing inode info to the same blocks). -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Question] Btrfs on iSCSI device
On Fri, 27 Jun 2014 18:34:34 Goffredo Baroncelli wrote: I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem. If you use a clustered filesystem then you can safely mount it on multiple machines. If you use a non-clustered filesystem it can still mount and even appear to work for a while. It's surprising how many writes you can make to a dual- mounted filesystem that's not designed for such things before you get a totally broken filesystem. On Fri, 27 Jun 2014 13:15:16 Austin S Hemmelgarn wrote: The reason it appears to work when using iSCSI and not with directly connected parallel SCSI or SAS is that iSCSI doesn't provide low level hardware access. I've tried this with dual-attached FC and had no problems mounting. In what way is directly connected SCSI different from FC? -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Question] Btrfs on iSCSI device
On 06/27/2014 07:40 PM, Russell Coker wrote: On Fri, 27 Jun 2014 18:34:34 Goffredo Baroncelli wrote: I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem. If you use a clustered filesystem then you can safely mount it on multiple machines. If you use a non-clustered filesystem it can still mount and even appear to work for a while. It's surprising how many writes you can make to a dual- mounted filesystem that's not designed for such things before you get a totally broken filesystem. On Fri, 27 Jun 2014 13:15:16 Austin S Hemmelgarn wrote: The reason it appears to work when using iSCSI and not with directly connected parallel SCSI or SAS is that iSCSI doesn't provide low level hardware access. I've tried this with dual-attached FC and had no problems mounting. In what way is directly connected SCSI different from FC? FC is actually it's own networking stack (and you can even run (in theory) other protocols like IP and ATM on top of it), whereas parallel SCSI is just a multi-drop bus, and SAS is just a tree-structured bus with point-to-point communications emulated on top of it. In other words, parallel SCSI has topological constraints like RS-485, SAS has topology constraints like USB, and FC has topology constraints like Ethernet. Secondarily, most filesystems on Linux will let you mount them multiple times on separate hosts (ext4 has features to prevent this, but they are expensive and therefore turned off by default, I think XFS might have similar features, but I'm not sure). BTRFS should in theory be more resilient than most because of the COW nature (as long as it's only a few commit cycles, you should still be able to recover most of the data just fine). smime.p7s Description: S/MIME Cryptographic Signature
