On 2015/04/20 13:08, Tsutomu Itoh wrote:
On 2015/04/20 5:15, Joel Best wrote:
Hi all, We have recently had major issues with our large btrfs volume crashing
and remounting read-only because it thinks it's out of space. The volume is
55TB on h/w raid 6 with 44TB free and the server is running Ubuntu 14.04 server
x64. The problem first happened with kernel 3.13 but I've since upgraded to
3.18 while trying to resolve this issue. The volume was first created with
kernel 3.13 and btfs-progs 3.12.
When this problem first cropped up, it crashed the kernel with the following
error:
BTRFS debug (device sda1): run_one_delayed_ref returned -28
Apr 17 19:49:28 NAS1 kernel: [189102.821859] BTRFS error (device sda1) in
btrfs_run_delayed_refs:2730: errno=-28 No space left
Apr 17 19:49:28 NAS1 kernel: [189102.821861] BTRFS info (device sda1):
forced readonly
Apr 17 19:49:28 NAS1 kernel: [189102.916132] btrfs: Transaction aborted
(error -28)
After some reading, it seemed like mounting with clear_cache to clear the disk
caching but the problem recurred a short time later. We then tried a balance,
and and fsck/fsck --repair which both failed to resolve the issue. Finally, we
decided to upgrade kernels from 3.13 to 3.18.
To try and reproduce the issue after the upgrade, I created a script which uses
fallocate to generate 1000 1GB files, deletes them, and repeats:
#!/bin/bash
while [ 1 ] ; do
echo "Creating 1000 files..."
i=0
while [ 1 ] ; do
fallocate -l 1G test.${i}
(( i++ ))
if [[ "$i" == "1000" ]] ; then
break
fi
done
echo "Removing Files..."
rm -f test.*
done
After a few successful iterations, this happened:
# /root/April2015-storage-failure/stress-fallocate.sh
Creating 1000 files...
fallocate: test.147: fallocate failed: No space left on device
fallocate: test.148: fallocate failed: No space left on device
fallocate: test.149: fallocate failed: No space left on device
fallocate: test.150: fallocate failed: No space left on device
fallocate: test.151: fallocate failed: No space left on device
fallocate: test.152: fallocate failed: No space left on device
fallocate: test.153: fallocate failed: No space left on device
fallocate: test.154: fallocate failed: No space left on device
fallocate: test.155: fallocate failed: No space left on device
fallocate: test.156: fallocate failed: No space left on device
^C
I think this problem to be fixed by the following patches.
However, it is not in the Linus's master tree yet.
Forrest Liu's patch:
[PATCH] Btrfs: fix find_free_dev_extent() malfunction in case device tree
has hole
http://marc.info/?l=linux-btrfs&m=142286232508765&w=2
Oops, following patch is newer.
[PATCH v3] Btrfs: fix find_free_dev_extent() malfunction in case device tree
has hole
http://marc.info/?l=linux-btrfs&m=142347427506763&w=2
Thanks,
Tsutomu
Zhao Lei's patch:
[PATCH v2 0/9] btrfs: Fix no_space on dd and rm loop
http://marc.info/?l=linux-btrfs&m=142855419528331&w=2
Thanks,
Tsutomu
There was no kernel crash this time. All btrfs tools show lots of space
available:
root@NAS1:/mlrg/tmp# btrfs fi usage /mlrg
Overall:
Device size: 54.56TiB
Device allocated: 11.31TiB
Device unallocated: 43.26TiB
Used: 11.30TiB
Free (estimated): 43.26TiB (min: 43.26TiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:11.26TiB, Used:11.26TiB
/dev/sdc1 11.26TiB
Metadata,single: Size:48.01GiB, Used:46.29GiB
/dev/sdc1 48.01GiB
System,single: Size:32.00MiB, Used:1.20MiB
/dev/sdc1 32.00MiB
Unallocated:
/dev/sdc1 43.26TiB
I'm not sure if this is expected behaviour with fallocate or a bug. It's the
only way I've found to reliably reproduce the problem (aside from making the
server available to my users).
Also, I first inadvertently ran the fallocate script when a scrub was running I
experienced a different crash... I'm not sure if it's related:
[12996.654120] kernel BUG at /home/kernel/COD/linux/fs/btrfs/inode.c:3123!
[12996.656776] invalid opcode: 0000 [#1] SMP
[12996.658473] Modules linked in: nfsv3 ip6t_REJECT nf_reject_ipv6 xt_hl
ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 x86_pkg_temp_thermal intel_powerclamp
ipt_REJECT coretemp ipmi_devintf nf_reject_ipv4 xt_limit xt_tcpudp kvm
xt_addrtype crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
iptable_filter ip_tables x_tables sb_edac lpc_ich edac_core mei_me mei ioatdma
shpchp wmi ipmi_si ipmi_msghandler 8250_fintek mac_hid megaraid lp parport
rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl
nfs lockd grace sunrpc fscache btrfs raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq igb isci i2c_algo_bit
raid1 hid_generic dca raid0 usbhid ptp ses libsas ahci multipath enclosure hid
libahci pps_core scsi_transport_sas megaraid_sas linear
[12996.697253] CPU: 5 PID: 10458 Comm: btrfs-cleaner Tainted: G C
3.18.11-031811-generic #201504041535
[12996.701338] Hardware name: Intel Corporation S2600CP/S2600CP, BIOS
SE5C600.86B.02.03.0003.041920141333 04/19/2014
[12996.705495] task: ffff881fdaafda00 ti: ffff881a49df4000 task.ti:
ffff881a49df4000
[12996.708514] RIP: 0010:[<ffffffffc03370a9>] [<ffffffffc03370a9>]
btrfs_orphan_add+0x1a9/0x1c0 [btrfs]
[12996.712306] RSP: 0018:ffff881a49df7c98 EFLAGS: 00010286
[12996.714429] RAX: 00000000ffffffe4 RBX: ffff880fd75f8000 RCX:
0000000000000000
[12996.717308] RDX: 0000000000002b12 RSI: 0000000000040000 RDI:
ffff880f5a51c138
[12996.791256] RBP: ffff881a49df7cd8 R08: ffffe8fffee20850 R09:
ffff881aa38d5d40
[12996.866007] R10: 0000000000000000 R11: 0000000000000010 R12:
ffff881fe4608dc0
[12996.941445] R13: ffff881c1f61d790 R14: ffff880fd75f8458 R15:
0000000000000001
[12997.016606] FS: 0000000000000000(0000) GS:ffff881ffe620000(0000)
knlGS:0000000000000000
[12997.163897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12997.238138] CR2: 000000000128d008 CR3: 0000000001c16000 CR4:
00000000001407e0
[12997.312295] Stack:
[12997.383946] ffff881a49df7cd8 ffffffffc0375e0f ffff881fd4b35800
ffff880080ad8200
[12997.528170] ffff881fd4b35800 ffff881aa38d5d40 ffff881fe4608dc0
0000000000000001
[12997.672790] ffff881a49df7d58 ffffffffc031f2c0 ffff880f5a51c000
00000004c0305ffa
[12997.816858] Call Trace:
[12997.886473] [<ffffffffc0375e0f>] ? lookup_free_space_inode+0x4f/0x100
[btrfs]
[12998.025910] [<ffffffffc031f2c0>] btrfs_remove_block_group+0x140/0x490
[btrfs]
[12998.166112] [<ffffffffc0359f55>] btrfs_remove_chunk+0x245/0x380 [btrfs]
[12998.238039] [<ffffffffc031f846>] btrfs_delete_unused_bgs+0x236/0x270
[btrfs]
[12998.309001] [<ffffffffc0328bfc>] cleaner_kthread+0x12c/0x190 [btrfs]
[12998.378869] [<ffffffffc0328ad0>] ?
btree_readpage_end_io_hook+0x2c0/0x2c0 [btrfs]
[12998.514511] [<ffffffff81093bc9>] kthread+0xc9/0xe0
[12998.581913] [<ffffffff81093b00>] ? flush_kthread_worker+0x90/0x90
[12998.648486] [<ffffffff817b54d8>] ret_from_fork+0x58/0x90
[12998.714302] [<ffffffff81093b00>] ? flush_kthread_worker+0x90/0x90
All data seems to be in tact, but the system is unusable due to the frequent
crashes. Does anyone have any suggestions on how to proceed? I've tried a
balance (crashed after a long time), scrub (no errors), and fsck to no avail.
Thanks for any help!
-Joel
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
