Hello, we use btrfs subvolumes for rsync-based backups. During backups btrfs often fails with "No space left" error and goes to readonly mode (dmesg output is below) while there's still plenty of unallocated space:
$ btrfs fi df /backup Data, single: total=15.75TiB, used=15.72TiB System, DUP: total=8.00MiB, used=1.91MiB Metadata, DUP: total=148.00GiB, used=146.20GiB GlobalReserve, single: total=512.00MiB, used=0.00B $ btrfs fi show /dev/md2 Label: none uuid: 32892e65-f78d-45a3-a7c4-980fedc14e63 Total devices 1 FS bytes used 15.86TiB devid 1 size 21.83TiB used 16.03TiB path /dev/md2 $ btrfs file usage /backup Overall: Device size: 21.83TiB Device allocated: 16.02TiB Device unallocated: 5.81TiB Device missing: 0.00B Used: 15.94TiB Free (estimated): 5.89TiB (min: 2.98TiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 296.64MiB) Data,single: Size:15.73TiB, Used:15.65TiB /dev/md2 15.73TiB Metadata,DUP: Size:148.00GiB, Used:146.07GiB /dev/md2 296.00GiB System,DUP: Size:8.00MiB, Used:1.91MiB /dev/md2 16.00MiB Unallocated: /dev/md2 5.81TiB It usually helps to rebalance 100% of metadata but the error reappears again after few days or weeks. I also tried "btrfs check --repair" but it requires approx. 45 GB of RAM/swap and crashes after several days of swapping. Btrfs runs on top of a single MD RAID1 device and is mounted with the following options: $ cat /proc/mounts /dev/md2 /backup btrfs rw,noatime,compress=lzo,space_cache,clear_cache,enospc_debug,subvolid=5,subvol=/ 0 0 Kernel version: 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.3-7~bpo8+1 (2016-01-19) x86_64 GNU/Linux (jessie-backports) [2151517.510044] BTRFS info (device md2): disk space caching is enabled [2151517.510047] BTRFS: has skinny extents [2266753.904426] use_block_rsv: 307 callbacks suppressed [2266753.904430] ------------[ cut here ]------------ [2266753.904453] WARNING: CPU: 7 PID: 17513 at /build/linux-kTc2b3/linux-4.3.3/fs/btrfs/extent-tree.c:7637 btrfs_alloc_tree_block+0x107/0x480 [btrfs]() [2266753.904481] BTRFS: block rsv returned -28 [2266753.904483] Modules linked in: binfmt_misc xt_comment xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 iptable_filter xt_conntrack iptable_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6table_mangle ip6table_raw iptable_mangle ip6_tables ip_tables x_tables nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc intel_powerc lamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support sha256_ssse3 sha256_generic hmac drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ast ttm drm_kms_helper drm i2c_ismt i2c_i801 joydev evdev tpm_tis ipmi_si tpm serio_raw acpi_cpufreq ipmi_msghandler 8250_fintek lpc_ich mfd_core shpchp pcspkr processor button autofs4 xfs libcrc32c btrfs xor raid6_pq dm_mod [2266753.904552] raid10 raid1 hid_generic usbhid hid md_mod sg sd_mod ahci libahci crc32c_intel ehci_pci mpt2sas ehci_hcd raid_class libata scsi_transport_sas igb i2c_algo_bit usbcore dca ptp usb_common scsi_mod pps_core [2266753.904574] CPU: 7 PID: 17513 Comm: kworker/u16:10 Tainted: G W 4.3.0-0.bpo.1-amd64 #1 Debian 4.3.3-7~bpo8+1 [2266753.904576] Hardware name: Supermicro SSG-5018A-AR12L/A1SA7, BIOS 1.0a 07/09/2014 [2266753.904597] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [2266753.904600] 0000000000000000 00000000071448ef ffffffff812e1889 ffff880003637868 [2266753.904604] ffffffff81074451 ffff880265c3e000 ffff8800036378c0 0000000000004000 [2266753.904608] ffff880339498970 0000000000000001 ffffffff810744dc ffffffffa0341c18 [2266753.904612] Call Trace: [2266753.904620] [<ffffffff812e1889>] ? dump_stack+0x40/0x57 [2266753.904625] [<ffffffff81074451>] ? warn_slowpath_common+0x81/0xb0 [2266753.904629] [<ffffffff810744dc>] ? warn_slowpath_fmt+0x5c/0x80 [2266753.904643] [<ffffffffa02b06d7>] ? btrfs_alloc_tree_block+0x107/0x480 [btrfs] [2266753.904649] [<ffffffff8101472c>] ? __switch_to+0x25c/0x590 [2266753.904662] [<ffffffffa0298645>] ? __btrfs_cow_block+0x145/0x5e0 [btrfs] [2266753.904674] [<ffffffffa0298c6f>] ? btrfs_cow_block+0x10f/0x1b0 [btrfs] [2266753.904687] [<ffffffffa029c86d>] ? btrfs_search_slot+0x1fd/0xa30 [btrfs] [2266753.904705] [<ffffffffa02deefd>] ? insert_state+0xbd/0x130 [btrfs] [2266753.904718] [<ffffffffa02a2f5e>] ? lookup_inline_extent_backref+0xee/0x650 [btrfs] [2266753.904723] [<ffffffff8116b801>] ? __set_page_dirty_nobuffers+0xe1/0x140 [2266753.904728] [<ffffffff811b76dc>] ? kmem_cache_alloc+0x21c/0x440 [2266753.904741] [<ffffffffa02a5bcd>] ? __btrfs_free_extent.isra.66+0x11d/0xd60 [btrfs] [2266753.904754] [<ffffffffa02a5887>] ? update_block_group.isra.65+0x127/0x350 [btrfs] [2266753.904773] [<ffffffffa030bde6>] ? btrfs_merge_delayed_refs+0x66/0x5e0 [btrfs] [2266753.904787] [<ffffffffa02a9e31>] ? __btrfs_run_delayed_refs+0x8b1/0x1080 [btrfs] [2266753.904801] [<ffffffffa02ad2c8>] ? btrfs_run_delayed_refs+0x78/0x2b0 [btrfs] [2266753.904815] [<ffffffffa02ad532>] ? delayed_ref_async_start+0x32/0x80 [btrfs] [2266753.904833] [<ffffffffa02f288c>] ? normal_work_helper+0xbc/0x240 [btrfs] [2266753.904837] [<ffffffff8108c53a>] ? process_one_work+0x14a/0x3d0 [2266753.904841] [<ffffffff8108cf75>] ? worker_thread+0x65/0x460 [2266753.904844] [<ffffffff8108cf10>] ? rescuer_thread+0x310/0x310 [2266753.904847] [<ffffffff8109222f>] ? kthread+0xdf/0x100 [2266753.904851] [<ffffffff81092150>] ? kthread_park+0x50/0x50 [2266753.904856] [<ffffffff8158a79f>] ? ret_from_fork+0x3f/0x70 [2266753.904860] [<ffffffff81092150>] ? kthread_park+0x50/0x50 [2266753.904862] ---[ end trace 42f58946d98c8b1f ]--- [2266753.904870] ------------[ cut here ]------------ [2266753.904884] WARNING: CPU: 7 PID: 17513 at /build/linux-kTc2b3/linux-4.3.3/fs/btrfs/extent-tree.c:6362 __btrfs_free_extent.isra.66+0x15b/0xd60 [btrfs]() [2266753.904886] BTRFS: Transaction aborted (error -28) [2266753.904888] Modules linked in: binfmt_misc [2266753.904894] BTRFS: error (device md2) in __btrfs_free_extent:6362: errno=-28 No space left [2266753.904898] BTRFS info (device md2): forced readonly [2266753.904900] BTRFS: error (device md2) in btrfs_run_delayed_refs:2858: errno=-28 No space left [2266753.905033] xt_comment xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 iptable_filter xt_conntrack iptable_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6table_mangle ip6table_raw iptable_mangle ip6_tables ip_tables x_tables nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc intel_powerclamp coretemp kvm_intel kvm cr ct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support sha256_ssse3 sha256_generic hmac drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ast ttm drm_kms_helper drm i2c_ismt i2c_i801 joydev evdev tpm_tis ipmi_si tpm serio_raw acpi_cpufreq ipmi_msghandler 8250_fintek lpc_ich mfd_core shpchp pcspkr processor button autofs4 xfs libcrc32c btrfs xor raid6_pq dm_mod raid10 raid1 hid_generic [2266753.905083] usbhid hid md_mod sg sd_mod ahci libahci crc32c_intel ehci_pci mpt2sas ehci_hcd raid_class libata scsi_transport_sas igb i2c_algo_bit usbcore dca ptp usb_common scsi_mod pps_core [2266753.905098] CPU: 7 PID: 17513 Comm: kworker/u16:10 Tainted: G W 4.3.0-0.bpo.1-amd64 #1 Debian 4.3.3-7~bpo8+1 [2266753.905101] Hardware name: Supermicro SSG-5018A-AR12L/A1SA7, BIOS 1.0a 07/09/2014 [2266753.905119] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [2266753.905121] 0000000000000000 00000000071448ef ffffffff812e1889 ffff880003637b20 [2266753.905125] ffffffff81074451 00000eed78f28000 ffff880003637b78 ffff880265c3e000 [2266753.905129] ffff880468aa2000 0000000000000000 ffffffff810744dc ffffffffa03417e8 [2266753.905133] Call Trace: [2266753.905137] [<ffffffff812e1889>] ? dump_stack+0x40/0x57 [2266753.905140] [<ffffffff81074451>] ? warn_slowpath_common+0x81/0xb0 [2266753.905144] [<ffffffff810744dc>] ? warn_slowpath_fmt+0x5c/0x80 [2266753.905158] [<ffffffffa02a5c0b>] ? __btrfs_free_extent.isra.66+0x15b/0xd60 [btrfs] [2266753.905171] [<ffffffffa02a5887>] ? update_block_group.isra.65+0x127/0x350 [btrfs] [2266753.905189] [<ffffffffa030bde6>] ? btrfs_merge_delayed_refs+0x66/0x5e0 [btrfs] [2266753.905203] [<ffffffffa02a9e31>] ? __btrfs_run_delayed_refs+0x8b1/0x1080 [btrfs] [2266753.905217] [<ffffffffa02ad2c8>] ? btrfs_run_delayed_refs+0x78/0x2b0 [btrfs] [2266753.905231] [<ffffffffa02ad532>] ? delayed_ref_async_start+0x32/0x80 [btrfs] [2266753.905249] [<ffffffffa02f288c>] ? normal_work_helper+0xbc/0x240 [btrfs] [2266753.905253] [<ffffffff8108c53a>] ? process_one_work+0x14a/0x3d0 [2266753.905256] [<ffffffff8108cf75>] ? worker_thread+0x65/0x460 [2266753.905259] [<ffffffff8108cf10>] ? rescuer_thread+0x310/0x310 [2266753.905263] [<ffffffff8109222f>] ? kthread+0xdf/0x100 [2266753.905266] [<ffffffff81092150>] ? kthread_park+0x50/0x50 [2266753.905270] [<ffffffff8158a79f>] ? ret_from_fork+0x3f/0x70 [2266753.905273] [<ffffffff81092150>] ? kthread_park+0x50/0x50 [2266753.905276] ---[ end trace 42f58946d98c8b20 ]--- [2266753.905280] BTRFS: error (device md2) in __btrfs_free_extent:6362: errno=-28 No space left [2266753.905348] BTRFS: error (device md2) in btrfs_run_delayed_refs:2858: errno=-28 No space left [2266766.878029] pending csums is 786432 The same or similar error seems to be reported multiple times: https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg48061.html https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg47355.html Best regards Martin Svec -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html