On 2014/06/17 11:10, Tsutomu Itoh wrote:
> On 2014/06/17 9:47, Chris Mason wrote:
>> On 06/16/2014 07:57 PM, Tsutomu Itoh wrote:
>>> On 2014/06/17 8:52, Chris Mason wrote:
>>>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote:
>>>>> Hi Chris,
>>>>>
>>>>> On 2014/06/17 2:56, Chris Mason wrote:
>>>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote:
>>>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on
>>>>>>> 3.16-rc1.
>>>>>>>
>>>>>>
>>>>>> Did we recover, or was it stuck forever?
>>>>>
>>>>> The following messages are repeatedly output.
>>>>> And stuck forever.
>>>>>
>>>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s!
>>>>> [kworker/u25:4:5189]
>>>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s!
>>>>> [kworker/u25:9:5194]
>>>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s!
>>>>> [kworker/u25:15:5200]
>>>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s!
>>>>> [kworker/u25:7:5192]
>>>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s!
>>>>> [kworker/u26:3:3182]
>>>>
>>>> Can you please capture a stack trace from all the cpus?
>>
>> Very strange, please try to reproduce again, I'll dig through things here.
>
> I can reproduce it easily in my environment.
This is my reproducer.
===================
#! /bin/bash
export TEST_DIR=/test5
export TEST_DEV=/dev/sdb4
TEST_DEV2=/dev/sdc6
export SCRATCH_MNT=/test6
export SCRATCH_DEV=/dev/sdc5
export MKFS_OPTIONS=""
export MOUNT_OPTIONS=""
umount ${TEST_DIR} ${SCRATCH_MNT}
mkfs.btrfs -f ${TEST_DEV} ${TEST_DEV2}
for i in $(seq 1 10)
do
./check 'btrfs/042'
done
umount ${TEST_DIR} ${SCRATCH_MNT}
===================
Stack trace is here.
==================
[ 1631.835052] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u25:1:193]
[ 1631.835053] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u25:3:3174]
[ 1631.835074] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.835076] CPU: 1 PID: 3174 Comm: kworker/u25:3 Not tainted 3.16.0-rc1 #1
[ 1631.835076] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.835094] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.835094] task: ffff880073d60000 ti: ffff88007188c000 task.ti:
ffff88007188c000
[ 1631.835097] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1631.835098] RSP: 0018:ffff88007188fac8 EFLAGS: 00000206
[ 1631.835098] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000a
[ 1631.835099] RDX: 0000000000000039 RSI: 000000000000000a RDI: ffff8801356755b0
[ 1631.835099] RBP: ffff88007188fac8 R08: ffff880135675574 R09: 0000000000000000
[ 1631.835100] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1631.835100] R13: 0000000000000000 R14: ffff8800766d3000 R15: ffffffffa05800a1
[ 1631.835101] FS: 0000000000000000(0000) GS:ffff88007dc40000(0000)
knlGS:0000000000000000
[ 1631.835101] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.835102] CR2: 0000000001858000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.835102] Stack:
[ 1631.835103] ffff88007188fad8 ffffffff8166bcbc ffff88007188fb38
ffffffffa05da140
[ 1631.835104] 0000000000000069 0000000000000035 0000000000000000
ffff88007188fb58
[ 1631.835105] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.835105] Call Trace:
[ 1631.835109] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.835119] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1631.835124] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1631.835129] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.835134] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1631.835143] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1631.835150] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.835157] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.835162] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.835170] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.835178] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.835187] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.835196] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.835198] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.835199] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.835200] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.835201] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.835203] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.835204] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.835205] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.835215] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1631.849920] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.849942] CPU: 0 PID: 193 Comm: kworker/u25:1 Not tainted 3.16.0-rc1 #1
[ 1631.849943] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.849953] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.849954] task: ffff880036a40960 ti: ffff88003695c000 task.ti:
ffff88003695c000
[ 1631.849955] RIP: 0010:[<ffffffff810ba7c2>] [<ffffffff810ba7c2>]
queue_write_lock_slowpath+0x62/0x90
[ 1631.849957] RSP: 0018:ffff88003695fae0 EFLAGS: 00000206
[ 1631.849958] RAX: 0000000000000101 RBX: 00000000000000cf RCX: 0000000000000101
[ 1631.849959] RDX: 00000000000000ff RSI: 0000000000000283 RDI: ffff8801356755b0
[ 1631.849960] RBP: ffff88003695fae0 R08: ffff88003695fb18 R09: 0000000000000000
[ 1631.849961] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffffa059cf56
[ 1631.849962] R13: ffff88003695fa78 R14: 0000000000000001 R15: 0000000000000012
[ 1631.849963] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000)
knlGS:0000000000000000
[ 1631.849964] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.849965] CR2: 000000000173e000 CR3: 0000000001c0e000 CR4: 00000000000007f0
[ 1631.849966] Stack:
[ 1631.849967] ffff88003695faf0 ffffffff8166bc51 ffff88003695fb50
ffffffffa05da519
[ 1631.849968] 0000000000000000 ffff880036a40960 ffffffff810b4840
ffff88003695fb18
[ 1631.849970] ffff88003695fb18 ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.849971] Call Trace:
[ 1631.849974] [<ffffffff8166bc51>] _raw_write_lock+0x21/0x30
[ 1631.849984] [<ffffffffa05da519>] btrfs_tree_lock+0xc9/0x1d0 [btrfs]
[ 1631.849986] [<ffffffff810b4840>] ? abort_exclusive_wait+0xb0/0xb0
[ 1631.849992] [<ffffffffa057ae3b>] btrfs_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.849998] [<ffffffffa0580029>] btrfs_search_slot+0x8f9/0xa10 [btrfs]
[ 1631.850003] [<ffffffffa057aba9>] ? btrfs_set_path_blocking+0x39/0x80 [btrfs]
[ 1631.850009] [<ffffffffa0581970>] btrfs_insert_empty_items+0x70/0xc0 [btrfs]
[ 1631.850016] [<ffffffffa05973b9>] btrfs_csum_file_blocks+0x2d9/0x680 [btrfs]
[ 1631.850022] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.850030] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.850039] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.850048] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.850057] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.850059] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.850060] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.850062] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.850064] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.850066] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.850067] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.850069] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.850070] Code: 2e 0f 1f 84 00 00 00 00 00 f3 90 8b 17 84 d2 75 f8 89 d1
89 d0 83 c9 01 f0 0f b1 0f 39 d0 75 e9 ba ff 00 00 00 eb 04 66 90 f3 90 <8b> 07
83 f8 01 75 f7 f0 0f b1 17 83 f8 01 75 ee eb bc f3 90 0f
[ 1631.860046] BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u25:4:3175]
[ 1631.868742] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.868763] CPU: 3 PID: 3175 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1
[ 1631.868764] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.868774] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.868775] task: ffff880073d60960 ti: ffff88007bde0000 task.ti:
ffff88007bde0000
[ 1631.868776] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1631.868778] RSP: 0018:ffff88007bde3ac8 EFLAGS: 00000206
[ 1631.868779] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000007
[ 1631.868780] RDX: 0000000000000036 RSI: 0000000000000007 RDI: ffff8801356755b0
[ 1631.868781] RBP: ffff88007bde3ac8 R08: ffff880135675574 R09: 0000000000000000
[ 1631.868782] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1631.868783] R13: 0000000000000000 R14: ffff88007385f000 R15: ffffffffa05800a1
[ 1631.868784] FS: 0000000000000000(0000) GS:ffff88007dcc0000(0000)
knlGS:0000000000000000
[ 1631.868785] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.868786] CR2: 0000000000d6e000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.868787] Stack:
[ 1631.868788] ffff88007bde3ad8 ffffffff8166bcbc ffff88007bde3b38
ffffffffa05da140
[ 1631.868789] 0000000000000068 0000000000000035 0000000000000000
ffff88007bde3b58
[ 1631.868791] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.868792] Call Trace:
[ 1631.868795] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.868804] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1631.868810] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1631.868815] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.868821] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1631.868830] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1631.868837] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.868845] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.868850] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.868859] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.868867] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.868869] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80
[ 1631.868878] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.868887] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.868889] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.868891] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.868892] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.868894] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.868896] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.868898] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.868899] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.868900] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1631.872043] BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u25:6:3177]
[ 1631.880892] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.880913] CPU: 4 PID: 3177 Comm: kworker/u25:6 Not tainted 3.16.0-rc1 #1
[ 1631.880914] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.880924] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.880926] task: ffff880073d61c20 ti: ffff880073d70000 task.ti:
ffff880073d70000
[ 1631.880927] RIP: 0010:[<ffffffff810ba85c>] [<ffffffff810ba85c>]
queue_read_lock_slowpath+0x6c/0x90
[ 1631.880929] RSP: 0018:ffff880073d73ac8 EFLAGS: 00000202
[ 1631.880930] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000c
[ 1631.880931] RDX: 000000000000003b RSI: 000000000000000c RDI: ffff8801356755b0
[ 1631.880932] RBP: ffff880073d73ac8 R08: ffff880135675574 R09: 0000000000000000
[ 1631.880933] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1631.880934] R13: 0000000000000000 R14: ffff8800718af000 R15: ffffffffa05800a1
[ 1631.880935] FS: 0000000000000000(0000) GS:ffff88007dd00000(0000)
knlGS:0000000000000000
[ 1631.880936] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.880937] CR2: 0000000000c98000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.880938] Stack:
[ 1631.880938] ffff880073d73ad8 ffffffff8166bcbc ffff880073d73b38
ffffffffa05da140
[ 1631.880940] 0000000000000068 0000000000000035 0000000000000000
ffff880073d73b58
[ 1631.880941] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.880943] Call Trace:
[ 1631.880945] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.880955] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1631.880960] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1631.880966] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.880972] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1631.880981] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1631.880988] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.880996] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.881001] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.881010] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.881018] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.881021] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0
[ 1631.881029] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.881039] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.881040] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.881042] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.881043] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.881045] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.881047] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.881049] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.881051] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.881051] Code: 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75
0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 0f b6 47 04 <38> c2
75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c ff 74 f8 5d
[ 1631.884041] BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u25:5:3176]
[ 1631.893041] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.893063] CPU: 5 PID: 3176 Comm: kworker/u25:5 Not tainted 3.16.0-rc1 #1
[ 1631.893064] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.893074] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.893075] task: ffff880073d612c0 ti: ffff880073d5c000 task.ti:
ffff880073d5c000
[ 1631.893076] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1631.893078] RSP: 0018:ffff880073d5fac8 EFLAGS: 00000206
[ 1631.893079] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000009
[ 1631.893080] RDX: 0000000000000038 RSI: 0000000000000009 RDI: ffff8801356755b0
[ 1631.893081] RBP: ffff880073d5fac8 R08: ffff880135675574 R09: 0000000000000000
[ 1631.893082] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1631.893083] R13: 0000000000001000 R14: ffff88007c65a000 R15: ffffffffa05800a1
[ 1631.893084] FS: 0000000000000000(0000) GS:ffff88007dd40000(0000)
knlGS:0000000000000000
[ 1631.893085] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.893086] CR2: 0000000001c95000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.893087] Stack:
[ 1631.893088] ffff880073d5fad8 ffffffff8166bcbc ffff880073d5fb38
ffffffffa05da140
[ 1631.893089] 00000000000000cc 00000000000000a0 0000000000000000
ffff880073d5fb58
[ 1631.893091] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.893092] Call Trace:
[ 1631.893095] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.893104] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1631.893110] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1631.893115] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.893121] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1631.893130] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1631.893138] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.893145] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.893151] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.893159] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.893168] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.893169] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0
[ 1631.893178] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.893188] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.893189] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.893191] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.893192] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.893194] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.893196] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.893198] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.893199] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.893200] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1631.969022] BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u26:6:3182]
[ 1631.976495] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.976516] CPU: 7 PID: 3182 Comm: kworker/u26:6 Not tainted 3.16.0-rc1 #1
[ 1631.976517] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.976527] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.976528] task: ffff8800ba65e720 ti: ffff880134714000 task.ti:
ffff880134714000
[ 1631.976529] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1631.976531] RSP: 0018:ffff880134717ac8 EFLAGS: 00000202
[ 1631.976532] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000008
[ 1631.976533] RDX: 0000000000000037 RSI: 0000000000000008 RDI: ffff8801356755b0
[ 1631.976534] RBP: ffff880134717ac8 R08: ffff880135675574 R09: 0000000000000000
[ 1631.976535] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1631.976536] R13: 0000000000001000 R14: ffff880075cd5000 R15: ffffffffa05800a1
[ 1631.976537] FS: 0000000000000000(0000) GS:ffff88013fc40000(0000)
knlGS:0000000000000000
[ 1631.976538] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.976539] CR2: 0000000000715000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.976540] Stack:
[ 1631.976540] ffff880134717ad8 ffffffff8166bcbc ffff880134717b38
ffffffffa05da140
[ 1631.976542] 00000000000000ce 00000000000000a0 0000000000000000
ffff880134717b58
[ 1631.976544] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1631.976545] Call Trace:
[ 1631.976547] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.976557] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1631.976562] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1631.976568] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1631.976573] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1631.976583] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1631.976590] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.976598] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.976603] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.976611] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.976620] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.976622] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80
[ 1631.976630] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.976640] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.976641] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.976643] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.976644] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.976646] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.976648] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.976650] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.976651] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.976652] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1631.981019] BUG: soft lockup - CPU#8 stuck for 22s! [kworker/u26:3:3179]
[ 1631.989015] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1631.989036] CPU: 8 PID: 3179 Comm: kworker/u26:3 Not tainted 3.16.0-rc1 #1
[ 1631.989037] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1631.989047] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1631.989048] task: ffff8800b7b13840 ti: ffff880137774000 task.ti:
ffff880137774000
[ 1631.989049] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1631.989051] RSP: 0018:ffff880137777b08 EFLAGS: 00000202
[ 1631.989052] RAX: 0000000000000033 RBX: 0000000000000012 RCX: 0000000000000001
[ 1631.989053] RDX: 0000000000000035 RSI: 0000000000000004 RDI: ffff8801356755b0
[ 1631.989054] RBP: ffff880137777b08 R08: 0000000000000001 R09: ffff8801357a0490
[ 1631.989055] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880135675680
[ 1631.989056] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002
[ 1631.989057] FS: 0000000000000000(0000) GS:ffff88013fc80000(0000)
knlGS:0000000000000000
[ 1631.989058] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1631.989059] CR2: 0000000001945000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1631.989060] Stack:
[ 1631.989060] ffff880137777b18 ffffffff8166bcbc ffff880137777b38
ffffffffa05d9fe4
[ 1631.989062] 0000000000000001 ffffffffffffffe8 ffff880137777b58
ffffffffa057ac2a
[ 1631.989063] ffff8801357bee10 0000000000000000 ffff880137777bf8
ffffffffa057fe32
[ 1631.989065] Call Trace:
[ 1631.989067] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1631.989077] [<ffffffffa05d9fe4>] btrfs_clear_lock_blocking_rw+0x44/0x150
[btrfs]
[ 1631.989082] [<ffffffffa057ac2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs]
[ 1631.989088] [<ffffffffa057fe32>] btrfs_search_slot+0x702/0xa10 [btrfs]
[ 1631.989095] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1631.989102] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1631.989108] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1631.989116] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1631.989125] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1631.989127] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0
[ 1631.989135] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1631.989145] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1631.989147] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1631.989148] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1631.989150] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1631.989151] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1631.989153] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.989155] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1631.989157] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1631.989158] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1631.994016] BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u26:0:3178]
[ 1632.002726] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1632.002747] CPU: 9 PID: 3178 Comm: kworker/u26:0 Not tainted 3.16.0-rc1 #1
[ 1632.002748] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1632.002758] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1632.002759] task: ffff8800b7b15dc0 ti: ffff88013397c000 task.ti:
ffff88013397c000
[ 1632.002760] RIP: 0010:[<ffffffff810ba7a4>] [<ffffffff810ba7a4>]
queue_write_lock_slowpath+0x44/0x90
[ 1632.002762] RSP: 0018:ffff88013397fb30 EFLAGS: 00000286
[ 1632.002763] RAX: 00000000000000ff RBX: ffff880135675680 RCX: 0000000000000001
[ 1632.002764] RDX: 00000000000000ff RSI: 0000000137fb4000 RDI: ffff8801356756f0
[ 1632.002765] RBP: ffff88013397fb30 R08: 0000000000000001 R09: ffff8801357a0490
[ 1632.002766] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000086
[ 1632.002767] R13: ffffffffa059cf56 R14: ffff88013397fac0 R15: 0000000000000001
[ 1632.002768] FS: 0000000000000000(0000) GS:ffff88013fcc0000(0000)
knlGS:0000000000000000
[ 1632.002769] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1632.002770] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1632.002771] Stack:
[ 1632.002771] ffff88013397fb40 ffffffff8166bc51 ffff88013397fb58
ffffffffa05da279
[ 1632.002773] ffff880135765750 ffff88013397fbf8 ffffffffa057fb4c
0000000000000001
[ 1632.002774] ffff88013a9df320 ffff88013a9df1e0 ffff88013397fbc0
ffff88013a9df1e0
[ 1632.002776] Call Trace:
[ 1632.002778] [<ffffffff8166bc51>] _raw_write_lock+0x21/0x30
[ 1632.002788] [<ffffffffa05da279>] btrfs_try_tree_write_lock+0x29/0x80 [btrfs]
[ 1632.002793] [<ffffffffa057fb4c>] btrfs_search_slot+0x41c/0xa10 [btrfs]
[ 1632.002801] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1632.002808] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1632.002814] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1632.002822] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1632.002831] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1632.002839] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1632.002849] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1632.002851] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1632.002852] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1632.002854] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1632.002855] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1632.002857] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.002859] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1632.002861] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.002862] Code: 75 5b 8b 07 85 c0 75 21 ba ff 00 00 00 f0 0f b1 17 85 c0
75 14 66 90 80 47 04 01 5d c3 66 2e 0f 1f 84 00 00 00 00 00 f3 90 8b 17 <84> d2
75 f8 89 d1 89 d0 83 c9 01 f0 0f b1 0f 39 d0 75 e9 ba ff
[ 1632.006013] BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u26:1:88]
[ 1632.014778] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1632.014799] CPU: 10 PID: 88 Comm: kworker/u26:1 Not tainted 3.16.0-rc1 #1
[ 1632.014800] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1632.014810] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1632.014811] task: ffff88013bb68000 ti: ffff88013bb5c000 task.ti:
ffff88013bb5c000
[ 1632.014812] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1632.014814] RSP: 0018:ffff88013bb5fac8 EFLAGS: 00000202
[ 1632.014815] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000b
[ 1632.014816] RDX: 000000000000003a RSI: 000000000000000b RDI: ffff8801356755b0
[ 1632.014817] RBP: ffff88013bb5fac8 R08: ffff880135675574 R09: 0000000000000000
[ 1632.014818] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000
[ 1632.014819] R13: 0000000000000000 R14: ffff88013b9bd000 R15: ffffffffa05800a1
[ 1632.014820] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000)
knlGS:0000000000000000
[ 1632.014821] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1632.014822] CR2: 0000000000715000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1632.014823] Stack:
[ 1632.014824] ffff88013bb5fad8 ffffffff8166bcbc ffff88013bb5fb38
ffffffffa05da140
[ 1632.014825] 0000000000000069 0000000000000035 0000000000000000
ffff88013bb5fb58
[ 1632.014827] ffffffffa057881f ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1632.014828] Call Trace:
[ 1632.014830] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1632.014840] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1632.014845] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs]
[ 1632.014851] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1632.014856] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1632.014866] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180
[btrfs]
[ 1632.014873] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs]
[ 1632.014880] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs]
[ 1632.014886] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1632.014894] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1632.014903] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1632.014905] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0
[ 1632.014913] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1632.014923] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1632.014925] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1632.014926] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1632.014928] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1632.014929] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1632.014931] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.014933] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1632.014935] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.014936] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
[ 1632.018011] BUG: soft lockup - CPU#11 stuck for 22s! [kworker/u26:7:3183]
[ 1632.027027] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables
btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd
xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp
iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801
mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd
ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect
sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi
megaraid_sas i2c_core ata_piix
[ 1632.027048] CPU: 11 PID: 3183 Comm: kworker/u26:7 Not tainted 3.16.0-rc1 #1
[ 1632.027049] Hardware name: FUJITSU-SV PRIMERGY
/D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010
[ 1632.027059] Workqueue: btrfs-endio-write normal_work_helper [btrfs]
[ 1632.027061] task: ffff8800ba65b840 ti: ffff88013a714000 task.ti:
ffff88013a714000
[ 1632.027061] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>]
queue_read_lock_slowpath+0x68/0x90
[ 1632.027064] RSP: 0018:ffff88013a717a80 EFLAGS: 00000202
[ 1632.027065] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000006
[ 1632.027066] RDX: 0000000000000034 RSI: 0000000000000006 RDI: ffff8801356755b0
[ 1632.027066] RBP: ffff88013a717a80 R08: ffff880135675574 R09: 0000000000000000
[ 1632.027067] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1632.027068] R13: ffff8801355f0870 R14: ffffffff00000000 R15: ffff8801355f0870
[ 1632.027069] FS: 0000000000000000(0000) GS:ffff88013fd40000(0000)
knlGS:0000000000000000
[ 1632.027070] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1632.027071] CR2: 0000000000b55000 CR3: 0000000001c0e000 CR4: 00000000000007e0
[ 1632.027072] Stack:
[ 1632.027073] ffff88013a717a90 ffffffff8166bcbc ffff88013a717af0
ffffffffa05da140
[ 1632.027074] ffffffff810b4178 00000000000000cf ffff88013a717b48
ffffffffa057d07d
[ 1632.027076] ffff88013a717bc0 ffff880135675540 ffff8800b7117800
ffff880000000000
[ 1632.027077] Call Trace:
[ 1632.027080] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30
[ 1632.027090] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs]
[ 1632.027091] [<ffffffff810b4178>] ? __wake_up_common+0x58/0x90
[ 1632.027097] [<ffffffffa057d07d>] ?
generic_bin_search.constprop.35+0x8d/0x1a0 [btrfs]
[ 1632.027103] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs]
[ 1632.027108] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs]
[ 1632.027118] [<ffffffffa05c38ef>] ? free_extent_buffer+0x4f/0xb0 [btrfs]
[ 1632.027124] [<ffffffffa0582eb8>] btrfs_next_old_leaf+0x218/0x4b0 [btrfs]
[ 1632.027130] [<ffffffffa0583160>] btrfs_next_leaf+0x10/0x20 [btrfs]
[ 1632.027137] [<ffffffffa059744b>] btrfs_csum_file_blocks+0x36b/0x680 [btrfs]
[ 1632.027143] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 1632.027151] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs]
[ 1632.027160] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs]
[ 1632.027168] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 1632.027178] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs]
[ 1632.027180] [<ffffffff8108ab95>] process_one_work+0x175/0x430
[ 1632.027181] [<ffffffff8108b23b>] worker_thread+0x11b/0x520
[ 1632.027183] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0
[ 1632.027184] [<ffffffff81091712>] kthread+0xd2/0xf0
[ 1632.027186] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.027188] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0
[ 1632.027190] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40
[ 1632.027191] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1
07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6
47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c
>
> Thanks,
> Tsutomu
>
> ==========
> Message from syslogd@luna at Jun 17 11:06:33 ...
> kernel:[ 560.076282] BUG: soft lockup - CPU#0 stuck for 22s!
> [kworker/u25:1:193]
>
> Message from syslogd@luna at Jun 17 11:06:33 ...
> kernel:[ 560.076283] BUG: soft lockup - CPU#1 stuck for 22s!
> [kworker/u25:3:3174]
>
> Message from syslogd@luna at Jun 17 11:06:33 ...
> kernel:[ 560.101276] BUG: soft lockup - CPU#3 stuck for 22s!
> [kworker/u25:4:3175]
>
> Message from syslogd@luna at Jun 17 11:06:33 ...
> kernel:[ 560.113274] BUG: soft lockup - CPU#4 stuck for 22s!
> [kworker/u25:6:3177]
>
> Message from syslogd@luna at Jun 17 11:06:33 ...
> kernel:[ 560.125271] BUG: soft lockup - CPU#5 stuck for 22s!
> [kworker/u25:5:3176]
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
