Re: Intermittent no space errors
On Wed, Aug 04, 2010 at 07:21:00PM +0800, Yan, Zheng wrote: > > We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs > > unstable HEAD to plain 2.6.35. > > > > [sr...@backup01:.../.rmagic]# rm * > > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > > rm: cannot remove `WEEKLY.html': No space left on device > > rm: cannot remove `YEARLY_bar3d.png': No space left on device > > rm: cannot remove `YEARLY.html': No space left on device >... > > Aug ?3 18:44:44 backup01 kernel: [ cut here ] > > Aug ?3 18:44:44 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 > > btrfs_block_rsv_check+0x151/0x180() >... > > These warning is because btrfs in 2.6.35 reserves more metadata space > for internal use > than older kernel. Your FS is too full, btrfs can't reserve enough > metadata space. Hello! Is it possible that 2.6.33.2 btrfs has mucked up the on-disk stuff in a way that causes 2.6.35 to be unhappy? The file system in question was reported to be 85% full, according to "df". In the meantime, we've been having some other problems on 2.6.35; for example, rsync has been trying to append a block to a file for the past 5 days. The file system is reported as 45% full: [sr...@backup01:/root]# df -Pt btrfs /backup/bu000/vol05/ Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/bu000-vol05 3221225472 1429529324 1791696148 45% /backup/bu000/vol05 [sr...@backup01:/root]# btrfs files df /backup/bu000/vol05 Data: total=1.57TB, used=1.31TB Metadata: total=15.51GB, used=10.48GB System: total=12.00MB, used=192.00KB At some point today, the kernel also spat this out: BUG: soft lockup - CPU#3 stuck for 61s! [rsync:21903] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 CPU 3 Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Pid: 21903, comm: rsync Tainted: GW 2.6.35-hw #2 0NK937/PowerEdge 1950 RIP: 0010:[] [] iput+0x5d/0x70 RSP: 0018:8802c14abb48 EFLAGS: 0246 RAX: RBX: 8802c14abb58 RCX: 0003 RDX: RSI: 0002 RDI: 88007c075980 RBP: 8100a84e R08: 0001 R09: 8000 R10: 0002 R11: R12: ff66 R13: 81af04e0 R14: R15: 7fff FS: 7fd13bbb06e0() GS:880001cc() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: 02f5a108 CR3: 0001eb94a000 CR4: 06e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process rsync (pid: 21903, threadinfo 8802c14aa000, task 880080b04b00) Stack: 88007c075888 88007c0757b0 8802c14abb98 812d7439 <0> 81664cde 0001 04d8 4000 <0> 88042a708178 88042a708000 8802c14abc08 812c599c Call Trace: [] ? btrfs_start_one_delalloc_inode+0x129/0x160 [] ? _raw_spin_lock+0xe/0x20 [] ? shrink_delalloc+0x8c/0x130 [] ? btrfs_delalloc_reserve_metadata+0x189/0x190 [] ? file_update_time+0x11e/0x180 [] ? btrfs_delalloc_reserve_space+0x43/0x60 [] ? btrfs_file_aio_write+0x508/0x970 [] ? apic_timer_interrupt+0xe/0x20 [] ? do_sync_write+0xd1/0x120 [] ? poll_select_copy_remaining+0xf7/0x140 [] ? vfs_write+0xcb/0x1a0 [] ? sys_write+0x50/0x90 [] ? system_call_fastpath+0x16/0x1b Code: 00 01 00 00 48 c7 c2 a0 2c 10 81 48 8b 40 30 48 85 c0 74 12 48 8b 50 20 48 c7 c0 a0 2c 10 81 48 85 d2 48 0 Call Trace: [] ? btrfs_start_one_delalloc_inode+0x129/0x160 [] ? _raw_spin_lock+0xe/0x20 [] ? shrink_delalloc+0x8c/0x130 [] ? btrfs_delalloc_reserve_metadata+0x189/0x190 [] ? file_update_time+0x11e/0x180 [] ? btrfs_delalloc_reserve_space+0x43/0x60 [] ? btrfs_file_aio_write+0x508/0x970 [] ? apic_timer_interrupt+0xe/0x20 [] ? do_sync_write+0xd1/0x120 [] ? poll_select_copy_remaining+0xf7/0x140 [] ? vfs_write+0xcb/0x1a0 [] ? sys_write+0x50/0x90 [] ? system_call_fastpath+0x16/0x1b [sr...@backup01:/root]# ls -l /proc/21903/fd/1 lrwx-- 1 root root 64 2010-08-09 18:21 /proc/21903/fd/1 -> /backup/bu000/vol05/vg005_web11_backup/2010-08-04-17-00/64/54/.../customer file.mov.aYX4Js [sr...@backup01:/root]# ls -lL /proc/21903/fd/1 -rw--- 1 root root 977797120 2010-08-04 20:39 /proc/21903/fd/1 [sr...@backup01:/root]# ps auxw|grep rsync root 21903 73.2 0.0 12912 192 ?RAug04 5177:08 rsync -aHq --numeric-ids --exclude-from=/etc/backups/backup.exclude --delete --delete-excluded /storage/vg005/web11/64/54/ /backup/bu000/vol05/vg005_web11_backup/2010-08-04-17-00/64/54 In other words, the rsync has made no progress for 5 days (or at least the mtime hasn't changed since then). "perf top" still shows output like this, showing that btrfs is trying to btrfs_find_space_cluster all of the time: samples pcnt function DSO ___ _ __
Re: Intermittent no space errors
On Wed, Aug 4, 2010 at 8:24 AM, Simon Kirby wrote: > On Wed, Jul 28, 2010 at 08:31:10AM +0800, Yan, Zheng wrote: > >> On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff wrote: >> > Hello, >> > >> > I installed the git repo kernel and added some debug to the ENOSPC >> > returns. Unfortunately its still failing. If it helps any its bombing >> > out in btrfs_check_data_free_space() in extent-tree.c. Returning on >> > the ENOSPC at line 2959. >> > >> > Unfortunately that is the extent of my ability to debug a filesystem. :P >> >> This is really helpful, thank you very much. > > We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs > unstable HEAD to plain 2.6.35. > > [sr...@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `WEEKLY.html': No space left on device > rm: cannot remove `YEARLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sr...@backup01:.../.rmagic]# l > total 25 > drwxr-xr-x 1 1037300 1037300 108 2010-08-03 18:19 ./ > drwxr-xr-x 1 1037300 1037300 28 2009-11-07 05:57 ../ > -rw-r--r-- 1 1037300 1037300 5720 2010-04-23 13:39 WEEKLY_bar3d.png > -rw-r--r-- 1 1037300 1037300 11882 2010-04-23 13:39 WEEKLY.html > -rw-r--r-- 1 1037300 1037300 2998 2010-04-23 13:39 YEARLY_bar3d.png > -rw-r--r-- 1 1037300 1037300 3016 2010-04-23 13:39 YEARLY.html > [sr...@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sr...@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sr...@backup01:.../.rmagic]# rm * > [sr...@backup01:.../.rmagic]# > > [sr...@backup01:/root]# btrfs filesystem df /backup/bu001/vol04 > Data: total=2.55TB, used=2.20TB > Metadata: total=230.13GB, used=183.83GB > System: total=12.00MB, used=548.00KB > [sr...@backup01:/root]# df -P /backup/bu001/vol04 > Filesystem 1024-blocks Used Available Capacity Mounted on > /dev/mapper/bu001-vol04 3221225472 2742785400 478440072 86% > /backup/bu001/vol04 > [sr...@backup01:/root]# l /dev/mapper/bu001-vol04 > brw-rw 1 root disk 252, 10 2010-08-03 16:02 /dev/mapper/bu001-vol04 > [sr...@backup01:/root]# btrfs filesystem show /dev/dm-10 > failed to read /dev/sr0 > Label: none uuid: 0c55f5f4-b618-4ec2-9dbc-e3e70a901e1a > Total devices 1 FS bytes used 2.37TB > devid 1 size 3.00TB used 3.00TB path /dev/dm-10 > > Btrfs Btrfs v0.19 > > We're also seeing things like this in dmesg, which appear to be coming > from btrfs-cleaner cleaning some old snapshot: > > Aug 3 18:40:45 backup01 kernel: [ cut here ] > Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 > btrfs_block_rsv_check+0x151/0x180() > Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 > Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si > ipmi_msghandler aoe bnx2 > Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G > W 2.6.35-hw #1 > Aug 3 18:40:45 backup01 kernel: Call Trace: > Aug 3 18:40:45 backup01 kernel: [] ? > btrfs_block_rsv_check+0x151/0x180 > Aug 3 18:40:45 backup01 kernel: [] > warn_slowpath_common+0x80/0xd0 > Aug 3 18:40:45 backup01 kernel: [] > warn_slowpath_null+0x15/0x20 > Aug 3 18:40:45 backup01 kernel: [] > btrfs_block_rsv_check+0x151/0x180 > Aug 3 18:40:45 backup01 kernel: [] > btrfs_should_end_transaction+0x61/0x90 > Aug 3 18:40:45 backup01 kernel: [] > btrfs_drop_snapshot+0x21d/0x5f0 > Aug 3 18:40:45 backup01 kernel: [] ? schedule+0x3f2/0x750 > Aug 3 18:40:45 backup01 kernel: [] > btrfs_clean_old_snapshots+0x12a/0x160 > Aug 3 18:40:45 backup01 kernel: [] > cleaner_kthread+0x160/0x190 > Aug 3 18:40:45 backup01 kernel: [] ? > cleaner_kthread+0x0/0x190 > Aug 3 18:40:45 backup01 kernel: [] kthread+0x96/0xb0 > Aug 3 18:40:45 backup01 kernel: [] > kernel_thread_helper+0x4/0x10 > Aug 3 18:40:45 backup01 kernel: [] ? kthread+0x0/0xb0 > Aug 3 18:40:45 backup01 kernel: [] ? > kernel_thread_helper+0x0/0x10 > Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45d ]--- > Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved > 4497289216 freed 0 78598144 > Aug 3 18:40:45 backup01 kernel: [ cut here ] > Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 > btrfs_block_rsv_check+0x151/0x180() > Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 > Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si > ipmi_msghandler aoe bnx2 > Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G > W 2.6.35-hw #1 > Aug 3 18:40:45 backup01 kernel: Call Trace: > Aug 3 18:40:45 backup01 kernel: [] ? > map_extent_buffer+0xb0/0xc0 > Aug 3 18:40:45 backup01 kernel: [] ? > bt
Re: Intermittent no space errors
On Wed, Jul 28, 2010 at 08:31:10AM +0800, Yan, Zheng wrote: > On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff wrote: > > Hello, > > > > I installed the git repo kernel and added some debug to the ENOSPC > > returns. Unfortunately its still failing. If it helps any its bombing > > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > > the ENOSPC at line 2959. > > > > Unfortunately that is the extent of my ability to debug a filesystem. :P > > This is really helpful, thank you very much. We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs unstable HEAD to plain 2.6.35. [sr...@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `WEEKLY.html': No space left on device rm: cannot remove `YEARLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sr...@backup01:.../.rmagic]# l total 25 drwxr-xr-x 1 1037300 1037300 108 2010-08-03 18:19 ./ drwxr-xr-x 1 1037300 103730028 2009-11-07 05:57 ../ -rw-r--r-- 1 1037300 1037300 5720 2010-04-23 13:39 WEEKLY_bar3d.png -rw-r--r-- 1 1037300 1037300 11882 2010-04-23 13:39 WEEKLY.html -rw-r--r-- 1 1037300 1037300 2998 2010-04-23 13:39 YEARLY_bar3d.png -rw-r--r-- 1 1037300 1037300 3016 2010-04-23 13:39 YEARLY.html [sr...@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sr...@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sr...@backup01:.../.rmagic]# rm * [sr...@backup01:.../.rmagic]# [sr...@backup01:/root]# btrfs filesystem df /backup/bu001/vol04 Data: total=2.55TB, used=2.20TB Metadata: total=230.13GB, used=183.83GB System: total=12.00MB, used=548.00KB [sr...@backup01:/root]# df -P /backup/bu001/vol04 Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/bu001-vol04 3221225472 2742785400 478440072 86% /backup/bu001/vol04 [sr...@backup01:/root]# l /dev/mapper/bu001-vol04 brw-rw 1 root disk 252, 10 2010-08-03 16:02 /dev/mapper/bu001-vol04 [sr...@backup01:/root]# btrfs filesystem show /dev/dm-10 failed to read /dev/sr0 Label: none uuid: 0c55f5f4-b618-4ec2-9dbc-e3e70a901e1a Total devices 1 FS bytes used 2.37TB devid1 size 3.00TB used 3.00TB path /dev/dm-10 Btrfs Btrfs v0.19 We're also seeing things like this in dmesg, which appear to be coming from btrfs-cleaner cleaning some old snapshot: Aug 3 18:40:45 backup01 kernel: [ cut here ] Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1 Aug 3 18:40:45 backup01 kernel: Call Trace: Aug 3 18:40:45 backup01 kernel: [] ? btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [] warn_slowpath_common+0x80/0xd0 Aug 3 18:40:45 backup01 kernel: [] warn_slowpath_null+0x15/0x20 Aug 3 18:40:45 backup01 kernel: [] btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [] btrfs_should_end_transaction+0x61/0x90 Aug 3 18:40:45 backup01 kernel: [] btrfs_drop_snapshot+0x21d/0x5f0 Aug 3 18:40:45 backup01 kernel: [] ? schedule+0x3f2/0x750 Aug 3 18:40:45 backup01 kernel: [] btrfs_clean_old_snapshots+0x12a/0x160 Aug 3 18:40:45 backup01 kernel: [] cleaner_kthread+0x160/0x190 Aug 3 18:40:45 backup01 kernel: [] ? cleaner_kthread+0x0/0x190 Aug 3 18:40:45 backup01 kernel: [] kthread+0x96/0xb0 Aug 3 18:40:45 backup01 kernel: [] kernel_thread_helper+0x4/0x10 Aug 3 18:40:45 backup01 kernel: [] ? kthread+0x0/0xb0 Aug 3 18:40:45 backup01 kernel: [] ? kernel_thread_helper+0x0/0x10 Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45d ]--- Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved 4497289216 freed 0 78598144 Aug 3 18:40:45 backup01 kernel: [ cut here ] Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1 Aug 3 18:40:45 backup01 kernel: Call Trace: Aug 3 18:40:45 backup01 kernel: [] ? map_extent_buffer+0xb0/0xc0 Aug 3 18:40:45 backup01 kernel: [] ? btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [] warn_slowpath_common+0x80/0xd0 Aug 3 18:40:45 backup01 kernel: [] warn_slowpath_null+0x15/0x20 Aug 3 18:40:45 backup01 kernel: [] btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:4
Re: Intermittent no space errors
Hello, Pherhaps it would be a good idea to add a tracepoint before each return ENOSPC? It shouldn't matter too much since a reasonable assumption would be that filesystems aren't running out of space most of the time. And you can use 'perf' for more insight in these cases without recompiling the kernel. Regards, justin On 27/07/10 22:30, Dave Cundiff wrote: > Hello, > > I installed the git repo kernel and added some debug to the ENOSPC > returns. Unfortunately its still failing. If it helps any its bombing > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > the ENOSPC at line 2959. > > Unfortunately that is the extent of my ability to debug a filesystem. :P > > Thanks, > > On Tue, Jul 27, 2010 at 9:19 AM, Yan, Zheng wrote: > >> On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff wrote: >> >>> Hello, >>> >>> On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I >>> have a backup process that fires up several rsync processes. These >>> mirror several dozen servers to individual sub-volumes. Everyday I >>> snapshot each sub-volume and rsync over it. >>> >>> The problem I'm seeing is my rsync processes are failing randomly with >>> "No space left on device". This is a 6 Terabyte volume with plenty of >>> free space. >>> >>> Mount options: >>> /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) >>> >>> [r...@rsync1 ~]# btrfs filesystem df /backups/ >>> Data: total=1.88TB, used=1.88TB >>> Metadata: total=43.38GB, used=32.06GB >>> System: total=12.00MB, used=260.00KB >>> >>> [r...@rsync1 ~]# df /dev/sdb >>> Filesystem 1K-blocks Used Available Use% Mounted on >>> /dev/sdb 5781249024 2087273084 3693975940 37% /backups >>> >>> They don't all fail at once. Normally I have 4-5 running at a time and >>> 1 or 2 will drop out with a no space error. The rest continue on. I've >>> noticed it will generally occur on ones that are in the middle of >>> transferring a very large file. If I lighten the load to one rsync at >>> a time it appears to happen less frequently. >>> >>> Any known issues I should be aware of? >>> >>> >> Thank you for reporting this. I will dig in. >> >> Yan, Zheng >> >> > > > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Intermittent no space errors
On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff wrote: > Hello, > > I installed the git repo kernel and added some debug to the ENOSPC > returns. Unfortunately its still failing. If it helps any its bombing > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > the ENOSPC at line 2959. > > Unfortunately that is the extent of my ability to debug a filesystem. :P > This is really helpful, thank you very much. Yan, Zheng -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Intermittent no space errors
Hello, I installed the git repo kernel and added some debug to the ENOSPC returns. Unfortunately its still failing. If it helps any its bombing out in btrfs_check_data_free_space() in extent-tree.c. Returning on the ENOSPC at line 2959. Unfortunately that is the extent of my ability to debug a filesystem. :P Thanks, On Tue, Jul 27, 2010 at 9:19 AM, Yan, Zheng wrote: > On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff wrote: >> Hello, >> >> On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I >> have a backup process that fires up several rsync processes. These >> mirror several dozen servers to individual sub-volumes. Everyday I >> snapshot each sub-volume and rsync over it. >> >> The problem I'm seeing is my rsync processes are failing randomly with >> "No space left on device". This is a 6 Terabyte volume with plenty of >> free space. >> >> Mount options: >> /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) >> >> [r...@rsync1 ~]# btrfs filesystem df /backups/ >> Data: total=1.88TB, used=1.88TB >> Metadata: total=43.38GB, used=32.06GB >> System: total=12.00MB, used=260.00KB >> >> [r...@rsync1 ~]# df /dev/sdb >> Filesystem 1K-blocks Used Available Use% Mounted on >> /dev/sdb 5781249024 2087273084 3693975940 37% /backups >> >> They don't all fail at once. Normally I have 4-5 running at a time and >> 1 or 2 will drop out with a no space error. The rest continue on. I've >> noticed it will generally occur on ones that are in the middle of >> transferring a very large file. If I lighten the load to one rsync at >> a time it appears to happen less frequently. >> >> Any known issues I should be aware of? >> > > Thank you for reporting this. I will dig in. > > Yan, Zheng > -- Dave Cundiff System Administrator A2Hosting, Inc http://www.a2hosting.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Intermittent no space errors
On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff wrote: > Hello, > > On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I > have a backup process that fires up several rsync processes. These > mirror several dozen servers to individual sub-volumes. Everyday I > snapshot each sub-volume and rsync over it. > > The problem I'm seeing is my rsync processes are failing randomly with > "No space left on device". This is a 6 Terabyte volume with plenty of > free space. > > Mount options: > /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) > > [r...@rsync1 ~]# btrfs filesystem df /backups/ > Data: total=1.88TB, used=1.88TB > Metadata: total=43.38GB, used=32.06GB > System: total=12.00MB, used=260.00KB > > [r...@rsync1 ~]# df /dev/sdb > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/sdb 5781249024 2087273084 3693975940 37% /backups > > They don't all fail at once. Normally I have 4-5 running at a time and > 1 or 2 will drop out with a no space error. The rest continue on. I've > noticed it will generally occur on ones that are in the middle of > transferring a very large file. If I lighten the load to one rsync at > a time it appears to happen less frequently. > > Any known issues I should be aware of? > Thank you for reporting this. I will dig in. Yan, Zheng -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Intermittent no space errors
Hello, On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I have a backup process that fires up several rsync processes. These mirror several dozen servers to individual sub-volumes. Everyday I snapshot each sub-volume and rsync over it. The problem I'm seeing is my rsync processes are failing randomly with "No space left on device". This is a 6 Terabyte volume with plenty of free space. Mount options: /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) [r...@rsync1 ~]# btrfs filesystem df /backups/ Data: total=1.88TB, used=1.88TB Metadata: total=43.38GB, used=32.06GB System: total=12.00MB, used=260.00KB [r...@rsync1 ~]# df /dev/sdb Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb 5781249024 2087273084 3693975940 37% /backups They don't all fail at once. Normally I have 4-5 running at a time and 1 or 2 will drop out with a no space error. The rest continue on. I've noticed it will generally occur on ones that are in the middle of transferring a very large file. If I lighten the load to one rsync at a time it appears to happen less frequently. Any known issues I should be aware of? Thanks, -- Dave Cundiff System Administrator A2Hosting, Inc http://www.a2hosting.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
