Hi all, We have recently had major issues with our large btrfs volume
crashing and remounting read-only because it thinks it's out of space.
The volume is 55TB on h/w raid 6 with 44TB free and the server is
running Ubuntu 14.04 server x64. The problem first happened with kernel
3.13 but I've since upgraded to 3.18 while trying to resolve this issue.
The volume was first created with kernel 3.13 and btfs-progs 3.12.
When this problem first cropped up, it crashed the kernel with the
following error:
BTRFS debug (device sda1): run_one_delayed_ref returned -28
Apr 17 19:49:28 NAS1 kernel: [189102.821859] BTRFS error (device
sda1) in btrfs_run_delayed_refs:2730: errno=-28 No space left
Apr 17 19:49:28 NAS1 kernel: [189102.821861] BTRFS info (device
sda1): forced readonly
Apr 17 19:49:28 NAS1 kernel: [189102.916132] btrfs: Transaction
aborted (error -28)
After some reading, it seemed like mounting with clear_cache to clear
the disk caching but the problem recurred a short time later. We then
tried a balance, and and fsck/fsck --repair which both failed to resolve
the issue. Finally, we decided to upgrade kernels from 3.13 to 3.18.
To try and reproduce the issue after the upgrade, I created a script
which uses fallocate to generate 1000 1GB files, deletes them, and repeats:
#!/bin/bash
while [ 1 ] ; do
echo "Creating 1000 files..."
i=0
while [ 1 ] ; do
fallocate -l 1G test.${i}
(( i++ ))
if [[ "$i" == "1000" ]] ; then
break
fi
done
echo "Removing Files..."
rm -f test.*
done
After a few successful iterations, this happened:
# /root/April2015-storage-failure/stress-fallocate.sh
Creating 1000 files...
fallocate: test.147: fallocate failed: No space left on device
fallocate: test.148: fallocate failed: No space left on device
fallocate: test.149: fallocate failed: No space left on device
fallocate: test.150: fallocate failed: No space left on device
fallocate: test.151: fallocate failed: No space left on device
fallocate: test.152: fallocate failed: No space left on device
fallocate: test.153: fallocate failed: No space left on device
fallocate: test.154: fallocate failed: No space left on device
fallocate: test.155: fallocate failed: No space left on device
fallocate: test.156: fallocate failed: No space left on device
^C
There was no kernel crash this time. All btrfs tools show lots of space
available:
root@NAS1:/mlrg/tmp# btrfs fi usage /mlrg
Overall:
Device size: 54.56TiB
Device allocated: 11.31TiB
Device unallocated: 43.26TiB
Used: 11.30TiB
Free (estimated): 43.26TiB (min: 43.26TiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:11.26TiB, Used:11.26TiB
/dev/sdc1 11.26TiB
Metadata,single: Size:48.01GiB, Used:46.29GiB
/dev/sdc1 48.01GiB
System,single: Size:32.00MiB, Used:1.20MiB
/dev/sdc1 32.00MiB
Unallocated:
/dev/sdc1 43.26TiB
I'm not sure if this is expected behaviour with fallocate or a bug. It's
the only way I've found to reliably reproduce the problem (aside from
making the server available to my users).
Also, I first inadvertently ran the fallocate script when a scrub was
running I experienced a different crash... I'm not sure if it's related:
[12996.654120] kernel BUG at
/home/kernel/COD/linux/fs/btrfs/inode.c:3123!
[12996.656776] invalid opcode: 0000 [#1] SMP
[12996.658473] Modules linked in: nfsv3 ip6t_REJECT nf_reject_ipv6
xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 x86_pkg_temp_thermal
intel_powerclamp ipt_REJECT coretemp ipmi_devintf nf_reject_ipv4
xt_limit xt_tcpudp kvm xt_addrtype crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper
ablk_helper cryptd nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
ip6table_filter ip6_tables nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
iptable_filter ip_tables x_tables sb_edac lpc_ich edac_core mei_me mei
ioatdma shpchp wmi ipmi_si ipmi_msghandler 8250_fintek mac_hid megaraid
lp parport rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace
sunrpc fscache btrfs raid10 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq igb isci i2c_algo_bit raid1
hid_generic dca raid0 usbhid ptp ses libsas ahci multipath enclosure hid
libahci pps_core scsi_transport_sas megaraid_sas linear
[12996.697253] CPU: 5 PID: 10458 Comm: btrfs-cleaner Tainted: G
C 3.18.11-031811-generic #201504041535
[12996.701338] Hardware name: Intel Corporation S2600CP/S2600CP,
BIOS SE5C600.86B.02.03.0003.041920141333 04/19/2014
[12996.705495] task: ffff881fdaafda00 ti: ffff881a49df4000 task.ti:
ffff881a49df4000
[12996.708514] RIP: 0010:[<ffffffffc03370a9>] [<ffffffffc03370a9>]
btrfs_orphan_add+0x1a9/0x1c0 [btrfs]
[12996.712306] RSP: 0018:ffff881a49df7c98 EFLAGS: 00010286
[12996.714429] RAX: 00000000ffffffe4 RBX: ffff880fd75f8000 RCX:
0000000000000000
[12996.717308] RDX: 0000000000002b12 RSI: 0000000000040000 RDI:
ffff880f5a51c138
[12996.791256] RBP: ffff881a49df7cd8 R08: ffffe8fffee20850 R09:
ffff881aa38d5d40
[12996.866007] R10: 0000000000000000 R11: 0000000000000010 R12:
ffff881fe4608dc0
[12996.941445] R13: ffff881c1f61d790 R14: ffff880fd75f8458 R15:
0000000000000001
[12997.016606] FS: 0000000000000000(0000)
GS:ffff881ffe620000(0000) knlGS:0000000000000000
[12997.163897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12997.238138] CR2: 000000000128d008 CR3: 0000000001c16000 CR4:
00000000001407e0
[12997.312295] Stack:
[12997.383946] ffff881a49df7cd8 ffffffffc0375e0f ffff881fd4b35800
ffff880080ad8200
[12997.528170] ffff881fd4b35800 ffff881aa38d5d40 ffff881fe4608dc0
0000000000000001
[12997.672790] ffff881a49df7d58 ffffffffc031f2c0 ffff880f5a51c000
00000004c0305ffa
[12997.816858] Call Trace:
[12997.886473] [<ffffffffc0375e0f>] ?
lookup_free_space_inode+0x4f/0x100 [btrfs]
[12998.025910] [<ffffffffc031f2c0>]
btrfs_remove_block_group+0x140/0x490 [btrfs]
[12998.166112] [<ffffffffc0359f55>] btrfs_remove_chunk+0x245/0x380
[btrfs]
[12998.238039] [<ffffffffc031f846>]
btrfs_delete_unused_bgs+0x236/0x270 [btrfs]
[12998.309001] [<ffffffffc0328bfc>] cleaner_kthread+0x12c/0x190
[btrfs]
[12998.378869] [<ffffffffc0328ad0>] ?
btree_readpage_end_io_hook+0x2c0/0x2c0 [btrfs]
[12998.514511] [<ffffffff81093bc9>] kthread+0xc9/0xe0
[12998.581913] [<ffffffff81093b00>] ? flush_kthread_worker+0x90/0x90
[12998.648486] [<ffffffff817b54d8>] ret_from_fork+0x58/0x90
[12998.714302] [<ffffffff81093b00>] ? flush_kthread_worker+0x90/0x90
All data seems to be in tact, but the system is unusable due to the
frequent crashes. Does anyone have any suggestions on how to proceed?
I've tried a balance (crashed after a long time), scrub (no errors), and
fsck to no avail.
Thanks for any help!
-Joel
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
