Re: experiences running btrfs on external USB disks?
On 2018-12-04 08:37, Graham Cobb wrote: On 04/12/2018 12:38, Austin S. Hemmelgarn wrote: In short, USB is _crap_ for fixed storage, don't use it like that, even if you are using filesystems which don't appear to complain. That's useful advice, thanks. Do you (or anyone else) have any experience of using btrfs over iSCSI? I was thinking about this for three different use cases: 1) Giving my workstation a data disk that is actually a partition on a server -- keeping all the data on the big disks on the server and reducing power consumption (just a small boot SSD in the workstation). 2) Splitting a btrfs RAID1 between a local disk and a remote iSCSI mirror to provide redundancy without putting more disks in the local system. Of course, this would mean that one of the RAID1 copies would have higher latency than the other. 3) Like case 1 but actually exposing an LVM logical volume from the server using iSCSI, rather than a simple disk partition. I would then put both encryption and RAID running on the server below that logical volume. NBD could also be an alternative to iSCSI in these cases as well. Any thoughts? I've not run it over iSCSI (I tend to avoid that overly-complicated mess), but I have done it over NBD and ATAoE, as well as some more exotic arrangements, and it's really not too bad. The important part is making sure your block layer and all the stuff under it are reliable, and USB is not.
Re: experiences running btrfs on external USB disks?
On 04/12/2018 12:38, Austin S. Hemmelgarn wrote: > In short, USB is _crap_ for fixed storage, don't use it like that, even > if you are using filesystems which don't appear to complain. That's useful advice, thanks. Do you (or anyone else) have any experience of using btrfs over iSCSI? I was thinking about this for three different use cases: 1) Giving my workstation a data disk that is actually a partition on a server -- keeping all the data on the big disks on the server and reducing power consumption (just a small boot SSD in the workstation). 2) Splitting a btrfs RAID1 between a local disk and a remote iSCSI mirror to provide redundancy without putting more disks in the local system. Of course, this would mean that one of the RAID1 copies would have higher latency than the other. 3) Like case 1 but actually exposing an LVM logical volume from the server using iSCSI, rather than a simple disk partition. I would then put both encryption and RAID running on the server below that logical volume. NBD could also be an alternative to iSCSI in these cases as well. Any thoughts? Graham
Re: experiences running btrfs on external USB disks?
On 2018-12-04 00:37, Tomasz Chmielewski wrote: I'm trying to use btrfs on an external USB drive, without much success. When the drive is connected for 2-3+ days, the filesystem gets remounted readonly, with BTRFS saying "IO failure": [77760.444607] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 [77760.550933] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: errno=-5 IO failure [77760.550979] BTRFS info (device sdb1): forced readonly [77760.551003] BTRFS: error (device sdb1) in btrfs_run_delayed_refs:2935: errno=-5 IO failure [77760.553223] BTRFS error (device sdb1): pending csums is 4096 Note that there are no other kernel messages (i.e. that would indicate a problem with disk, cable disconnection etc.). The load on the drive itself can be quite heavy at times (i.e. 100% IO for 1-2 h and more) - can it contribute to the problem (i.e. btrfs thinks there is some timeout somewhere)? Running 4.19.6 right now, but was experiencing the issue also with 4.18 kernels. # btrfs device stats /data [/dev/sda1].write_io_errs 0 [/dev/sda1].read_io_errs 0 [/dev/sda1].flush_io_errs 0 [/dev/sda1].corruption_errs 0 [/dev/sda1].generation_errs 0 It looks to me like the typical USB issues that are present with almost all filesystems but only seem to be noticed by BTRFS because it does more rigorous checking of data. In short, USB is _crap_ for fixed storage, don't use it like that, even if you are using filesystems which don't appear to complain.
Re: experiences running btrfs on external USB disks?
On 2018-12-04 14:59, Chris Murphy wrote: Running 4.19.6 right now, but was experiencing the issue also with 4.18 kernels. # btrfs device stats /data [/dev/sda1].write_io_errs0 [/dev/sda1].read_io_errs 0 [/dev/sda1].flush_io_errs0 [/dev/sda1].corruption_errs 0 [/dev/sda1].generation_errs 0 Hard to say without a complete dmesg; but errno=-5 IO failure is pretty much some kind of hardware problem in my experience. I haven't seen it be a bug. It is a complete dmesg - in sense: # grep -i btrfs -A5 -B5 /var/log/syslog Dec 4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed remove event for unknown device "/sys/skbuff_head_cache(1481:anacron.service)" Dec 4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed remove event for unknown device "/sys/buffer_head(1481:anacron.service)" Dec 4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed remove event for unknown device "/sys/ext4_inode_cache(1481:anacron.service)" Dec 4 05:15:01 step CRON[9352]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Dec 4 05:17:01 step CRON[9358]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Dec 4 05:23:13 step kernel: [77760.444607] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 Dec 4 05:23:13 step kernel: [77760.550933] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 Dec 4 05:23:13 step kernel: [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: errno=-5 IO failure Dec 4 05:23:13 step kernel: [77760.550979] BTRFS info (device sdb1): forced readonly Dec 4 05:23:13 step kernel: [77760.551003] BTRFS: error (device sdb1) in btrfs_run_delayed_refs:2935: errno=-5 IO failure Dec 4 05:23:13 step kernel: [77760.553223] BTRFS error (device sdb1): pending csums is 4096 Dec 4 05:23:14 step postfix/pickup[8993]: 13BBE460F86: uid=0 from= Dec 4 05:23:14 step postfix/cleanup[9398]: 13BBE460F86: message-id=<20181204052314.13BBE460F86@step> Dec 4 05:23:14 step postfix/qmgr[2745]: 13BBE460F86: from=, size=404, nrcpt=1 (queue active) Dec 4 05:23:14 step postfix/pickup[8993]: 40A964603EC: uid=0 from= [...some emails follow, usual CRON messages etc., but noting at all generated by the kernel, no hardware issue reported...] Tomasz Chmielewski
Re: experiences running btrfs on external USB disks?
On Mon, Dec 3, 2018 at 10:44 PM Tomasz Chmielewski wrote: > > I'm trying to use btrfs on an external USB drive, without much success. > > When the drive is connected for 2-3+ days, the filesystem gets remounted > readonly, with BTRFS saying "IO failure": > > [77760.444607] BTRFS error (device sdb1): bad tree block start, want > 378372096 have 0 > [77760.550933] BTRFS error (device sdb1): bad tree block start, want > 378372096 have 0 > [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: > errno=-5 IO failure > [77760.550979] BTRFS info (device sdb1): forced readonly > [77760.551003] BTRFS: error (device sdb1) in > btrfs_run_delayed_refs:2935: errno=-5 IO failure > [77760.553223] BTRFS error (device sdb1): pending csums is 4096 > > > Note that there are no other kernel messages (i.e. that would indicate a > problem with disk, cable disconnection etc.). > > The load on the drive itself can be quite heavy at times (i.e. 100% IO > for 1-2 h and more) - can it contribute to the problem (i.e. btrfs > thinks there is some timeout somewhere)? > > Running 4.19.6 right now, but was experiencing the issue also with 4.18 > kernels. > > > > # btrfs device stats /data > [/dev/sda1].write_io_errs0 > [/dev/sda1].read_io_errs 0 > [/dev/sda1].flush_io_errs0 > [/dev/sda1].corruption_errs 0 > [/dev/sda1].generation_errs 0 Hard to say without a complete dmesg; but errno=-5 IO failure is pretty much some kind of hardware problem in my experience. I haven't seen it be a bug. -- Chris Murphy
experiences running btrfs on external USB disks?
I'm trying to use btrfs on an external USB drive, without much success. When the drive is connected for 2-3+ days, the filesystem gets remounted readonly, with BTRFS saying "IO failure": [77760.444607] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 [77760.550933] BTRFS error (device sdb1): bad tree block start, want 378372096 have 0 [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: errno=-5 IO failure [77760.550979] BTRFS info (device sdb1): forced readonly [77760.551003] BTRFS: error (device sdb1) in btrfs_run_delayed_refs:2935: errno=-5 IO failure [77760.553223] BTRFS error (device sdb1): pending csums is 4096 Note that there are no other kernel messages (i.e. that would indicate a problem with disk, cable disconnection etc.). The load on the drive itself can be quite heavy at times (i.e. 100% IO for 1-2 h and more) - can it contribute to the problem (i.e. btrfs thinks there is some timeout somewhere)? Running 4.19.6 right now, but was experiencing the issue also with 4.18 kernels. # btrfs device stats /data [/dev/sda1].write_io_errs0 [/dev/sda1].read_io_errs 0 [/dev/sda1].flush_io_errs0 [/dev/sda1].corruption_errs 0 [/dev/sda1].generation_errs 0 Tomasz Chmielewski
