Hi Jim,
Have you tried to update the kernel as suggested by Changwei?
The messages seem to indicate kernel 4.4.0, a google search shows this
ubuntu version should be able to use kernel 4.10.
Best Regards,Luis
On Wednesday, January 10, 2018 4:12 PM, Jim Okken <j...@jokken.com> wrote:
hello again list,
We seem to be having issues on more servers where according to the linux
developers here: "the kernel is stuck in a spin lock during a disk operation."
The call traces are below, I see a lot of ocfs in the call traces, but I don't
know how to read them, please tell me does the issue come from ocfs?thanks --Jim
2018-01-06T17:10:02.194362+00:00 node-115 kernel: [87885.155288] Modules linked
in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp xt_physdev
br_netfilter veth ebtable_filter ebtables openvswitch ocfs2 quota_tree
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue
configfs ip6table_filter ip6_tables xt_multiport xt_conntrack iptable_filter
xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q garp mrp stp
llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm
lrw gf128mul glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek
serio_raw ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua
dm_multipath2018-01-06T17:10:02.194364+00:00 node-115 kernel: [87885.157143]
CPU: 15 PID: 11936 Comm: qemu-system-x86 Not tainted 4.4.0-98-generic
#121-Ubuntu2018-01-06T17:10:02.194366+00:00 node-115 kernel: [87885.157144]
Hardware name: HP ProLiant BL460c Gen9, BIOS I36
02/17/20172018-01-06T17:10:02.194367+00:00 node-115 kernel: [87885.157280]
task: ffff882036ff0000 ti: ffff881f80ca0000 task.ti:
ffff881f80ca00002018-01-06T17:10:02.194400+00:00 node-115 kernel:
[87885.157281] RIP: 0010:[<ffffffff810cb27c>] [<ffffffff810cb27c>]
native_queued_spin_lock_slowpath+0x15c/0x1702018-01-06T17:10:02.194414+00:00
node-115 kernel: [87885.157566] RSP: 0018:ffff88203f143c30 EFLAGS:
000002022018-01-06T17:10:02.194416+00:00 node-115 kernel: [87885.157567] RAX:
0000000000000101 RBX: ffff8820046c83f0 RCX:
00000000000000012018-01-06T17:10:02.194418+00:00 node-115 kernel:
[87885.157705] RDX: 0000000000000101 RSI: 0000000000000001 RDI:
ffff8820046c83ec2018-01-06T17:10:02.194440+00:00 node-115 kernel:
[87885.157705] RBP: ffff88203f143c30 R08: 0000000000000101 R09:
ffffffff811924a72018-01-06T17:10:02.194442+00:00 node-115 kernel:
[87885.157706] R10: ffffea0040d6d680 R11: 0000000000000800 R12:
ffff8820046c83ec2018-01-06T17:10:02.194443+00:00 node-115 kernel:
[87885.157707] R13: 0000000000000800 R14: 000000004c63ee00 R15:
00000000000008002018-01-06T17:10:02.194444+00:00 node-115 kernel:
[87885.157708] FS: 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000)
knlGS:00000000000000002018-01-06T17:10:02.194444+00:00 node-115 kernel:
[87885.157709] CS: 0010 DS: 0000 ES: 0000 CR0:
00000000800500332018-01-06T17:10:02.194445+00:00 node-115 kernel:
[87885.157710] CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4:
00000000001426e02018-01-06T17:10:02.194446+00:00 node-115 kernel:
[87885.157711] Stack:2018-01-06T17:10:02.194448+00:00 node-115 kernel:
[87885.157712] ffff88203f143c40 ffffffff81844421 ffff88203f143c60
ffffffff818425352018-01-06T17:10:02.194449+00:00 node-115 kernel:
[87885.157714] ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70
ffffffff8184257b2018-01-06T17:10:02.194450+00:00 node-115 kernel:
[87885.157716] ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80
00000000000008002018-01-06T17:10:02.194450+00:00 node-115 kernel:
[87885.157717] Call Trace:2018-01-06T17:10:02.194451+00:00 node-115 kernel:
[87885.157718] <IRQ>2018-01-06T17:10:02.194453+00:00 node-115 kernel:
[87885.157725] [<ffffffff81844421>]
_raw_spin_lock+0x21/0x302018-01-06T17:10:02.194454+00:00 node-115 kernel:
[87885.157727] [<ffffffff81842535>]
__mutex_unlock_slowpath+0x25/0x502018-01-06T17:10:02.194456+00:00 node-115
kernel: [87885.157729] [<ffffffff8184257b>]
mutex_unlock+0x1b/0x202018-01-06T17:10:02.194457+00:00 node-115 kernel:
[87885.157766] [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80
[ocfs2]2018-01-06T17:10:02.194458+00:00 node-115 kernel: [87885.157770]
[<ffffffff8124e57c>] dio_complete+0x11c/0x1c02018-01-06T17:10:02.194460+00:00
node-115 kernel: [87885.157771] [<ffffffff8124e693>]
dio_bio_end_aio+0x73/0x1002018-01-06T17:10:02.194461+00:00 node-115 kernel:
[87885.157774] [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:02.194463+00:00 node-115 kernel:
[87885.157777] [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:02.194464+00:00 node-115 kernel:
[87885.157780] [<ffffffff816bbd66>]
end_clone_bio+0x46/0x702018-01-06T17:10:02.194465+00:00 node-115 kernel:
[87885.157782] [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:02.194465+00:00 node-115 kernel:
[87885.157783] [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:02.194467+00:00 node-115 kernel:
[87885.157786] [<ffffffff815c52f3>]
scsi_end_request+0x33/0x1d02018-01-06T17:10:02.194468+00:00 node-115 kernel:
[87885.157788] [<ffffffff815c8a26>]
scsi_io_completion+0x1b6/0x6902018-01-06T17:10:02.194469+00:00 node-115 kernel:
[87885.157792] [<ffffffff810beb46>] ?
rebalance_domains+0x166/0x2d02018-01-06T17:10:02.194470+00:00 node-115 kernel:
[87885.157795] [<ffffffff815bf64f>]
scsi_finish_command+0xcf/0x1202018-01-06T17:10:02.194471+00:00 node-115 kernel:
[87885.157796] [<ffffffff815c81b4>]
scsi_softirq_done+0x124/0x1502018-01-06T17:10:02.194471+00:00 node-115 kernel:
[87885.157799] [<ffffffff813d3787>]
blk_done_softirq+0x87/0xb02018-01-06T17:10:02.194480+00:00 node-115 kernel:
[87885.157803] [<ffffffff81085dc1>]
__do_softirq+0x101/0x2902018-01-06T17:10:02.194481+00:00 node-115 kernel:
[87885.157805] [<ffffffff810860c3>]
irq_exit+0xa3/0xb02018-01-06T17:10:02.194483+00:00 node-115 kernel:
[87885.157809] [<ffffffff81050e93>]
smp_call_function_single_interrupt+0x33/0x402018-01-06T17:10:02.194483+00:00
node-115 kernel: [87885.157811] [<ffffffff81845ae2>]
call_function_single_interrupt+0x82/0x902018-01-06T17:10:02.194484+00:00
node-115 kernel: [87885.157812] <EOI>2018-01-06T17:10:02.194484+00:00 node-115
kernel: [87885.157814] [<ffffffff81844414>] ?
_raw_spin_lock+0x14/0x302018-01-06T17:10:02.194486+00:00 node-115 kernel:
[87885.157815] [<ffffffff81842422>]
__mutex_lock_slowpath+0x72/0x1302018-01-06T17:10:02.194487+00:00 node-115
kernel: [87885.157829] [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120
[ocfs2]2018-01-06T17:10:02.194488+00:00 node-115 kernel: [87885.157831]
[<ffffffff818424ff>] mutex_lock+0x1f/0x302018-01-06T17:10:02.194489+00:00
node-115 kernel: [87885.157843] [<ffffffffc076177a>]
ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]2018-01-06T17:10:02.194490+00:00
node-115 kernel: [87885.157847] [<ffffffff812252c0>] ?
poll_select_copy_remaining+0x140/0x1402018-01-06T17:10:02.194490+00:00 node-115
kernel: [87885.157858] [<ffffffffc0760e20>] ?
ocfs2_check_range_for_refcount+0x150/0x150
[ocfs2]2018-01-06T17:10:02.194492+00:00 node-115 kernel: [87885.157862]
[<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d02018-01-06T17:10:02.194493+00:00
node-115 kernel: [87885.157865] [<ffffffff8122e8e5>] ?
__fget_light+0x25/0x602018-01-06T17:10:02.194495+00:00 node-115 kernel:
[87885.157867] [<ffffffff8122e933>] ?
__fdget+0x13/0x202018-01-06T17:10:02.194496+00:00 node-115 kernel:
[87885.157868] [<ffffffff812622cf>]
do_io_submit+0x25f/0x5002018-01-06T17:10:02.194497+00:00 node-115 kernel:
[87885.157871] [<ffffffff81262580>]
SyS_io_submit+0x10/0x202018-01-06T17:10:02.194499+00:00 node-115 kernel:
[87885.157873] [<ffffffff818446b2>]
entry_SYSCALL_64_fastpath+0x16/0x712018-01-06T17:10:02.194500+00:00 node-115
kernel: [87885.157874] Code: 01 48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0
74 f6 c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90
<8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00
0f2018-01-06T17:10:30.192979+00:00 node-115 kernel: [87913.154413] Modules
linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport xt_conntrack
iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q
garp mrp stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
crct10dif_pclmul ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel
aesni_intel aes_x86_64 kvm lrw gf128mul glue_helper ablk_helper irqbypass
cryptd hpilo 8250_fintek serio_raw ioatdma ipmi_si sb_edac edac_core
ipmi_msghandler shpchp dca acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm
iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6
nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor dm_round_robin
ses enclosure scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath
linear uas usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel
scsi_transport_fc udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua
dm_multipath2018-01-06T17:10:30.192984+00:00 node-115 kernel: [87913.155150]
CPU: 15 PID: 11936 Comm: qemu-system-x86 Tainted: G L
4.4.0-98-generic #121-Ubuntu2018-01-06T17:10:30.192987+00:00 node-115 kernel:
[87913.155151] Hardware name: HP ProLiant BL460c Gen9, BIOS I36
02/17/20172018-01-06T17:10:30.192988+00:00 node-115 kernel: [87913.155153]
task: ffff882036ff0000 ti: ffff881f80ca0000 task.ti:
ffff881f80ca00002018-01-06T17:10:30.192990+00:00 node-115 kernel:
[87913.155154] RIP: 0010:[<ffffffff810cb27e>] [<ffffffff810cb27e>]
native_queued_spin_lock_slowpath+0x15e/0x1702018-01-06T17:10:30.192992+00:00
node-115 kernel: [87913.155160] RSP: 0018:ffff88203f143c30 EFLAGS:
000002022018-01-06T17:10:30.192994+00:00 node-115 kernel: [87913.155161] RAX:
0000000000000101 RBX: ffff8820046c83f0 RCX:
00000000000000012018-01-06T17:10:30.192996+00:00 node-115 kernel:
[87913.155162] RDX: 0000000000000101 RSI: 0000000000000001 RDI:
ffff8820046c83ec2018-01-06T17:10:30.193019+00:00 node-115 kernel:
[87913.155163] RBP: ffff88203f143c30 R08: 0000000000000101 R09:
ffffffff811924a72018-01-06T17:10:30.193023+00:00 node-115 kernel:
[87913.155164] R10: ffffea0040d6d680 R11: 0000000000000800 R12:
ffff8820046c83ec2018-01-06T17:10:30.193024+00:00 node-115 kernel:
[87913.155165] R13: 0000000000000800 R14: 000000004c63ee00 R15:
00000000000008002018-01-06T17:10:30.193026+00:00 node-115 kernel:
[87913.155166] FS: 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000)
knlGS:00000000000000002018-01-06T17:10:30.193028+00:00 node-115 kernel:
[87913.155167] CS: 0010 DS: 0000 ES: 0000 CR0:
00000000800500332018-01-06T17:10:30.193030+00:00 node-115 kernel:
[87913.155168] CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4:
00000000001426e02018-01-06T17:10:30.193032+00:00 node-115 kernel:
[87913.155169] Stack:2018-01-06T17:10:30.193034+00:00 node-115 kernel:
[87913.155170] ffff88203f143c40 ffffffff81844421 ffff88203f143c60
ffffffff818425352018-01-06T17:10:30.193036+00:00 node-115 kernel:
[87913.155172] ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70
ffffffff8184257b2018-01-06T17:10:30.193037+00:00 node-115 kernel:
[87913.155173] ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80
00000000000008002018-01-06T17:10:30.193039+00:00 node-115 kernel:
[87913.155175] Call Trace:2018-01-06T17:10:30.193040+00:00 node-115 kernel:
[87913.155176] <IRQ>2018-01-06T17:10:30.193042+00:00 node-115 kernel:
[87913.155183] [<ffffffff81844421>]
_raw_spin_lock+0x21/0x302018-01-06T17:10:30.193044+00:00 node-115 kernel:
[87913.155186] [<ffffffff81842535>]
__mutex_unlock_slowpath+0x25/0x502018-01-06T17:10:30.193046+00:00 node-115
kernel: [87913.155187] [<ffffffff8184257b>]
mutex_unlock+0x1b/0x202018-01-06T17:10:30.193047+00:00 node-115 kernel:
[87913.155224] [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80
[ocfs2]2018-01-06T17:10:30.193049+00:00 node-115 kernel: [87913.155228]
[<ffffffff8124e57c>] dio_complete+0x11c/0x1c02018-01-06T17:10:30.193051+00:00
node-115 kernel: [87913.155230] [<ffffffff8124e693>]
dio_bio_end_aio+0x73/0x1002018-01-06T17:10:30.193053+00:00 node-115 kernel:
[87913.155233] [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:30.193055+00:00 node-115 kernel:
[87913.155235] [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:30.193058+00:00 node-115 kernel:
[87913.155239] [<ffffffff816bbd66>]
end_clone_bio+0x46/0x702018-01-06T17:10:30.193060+00:00 node-115 kernel:
[87913.155240] [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:30.193062+00:00 node-115 kernel:
[87913.155242] [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:30.193064+00:00 node-115 kernel:
[87913.155245] [<ffffffff815c52f3>]
scsi_end_request+0x33/0x1d02018-01-06T17:10:30.193066+00:00 node-115 kernel:
[87913.155247] [<ffffffff815c8a26>]
scsi_io_completion+0x1b6/0x6902018-01-06T17:10:30.193068+00:00 node-115 kernel:
[87913.155251] [<ffffffff810beb46>] ?
rebalance_domains+0x166/0x2d02018-01-06T17:10:30.193069+00:00 node-115 kernel:
[87913.155254] [<ffffffff815bf64f>]
scsi_finish_command+0xcf/0x1202018-01-06T17:10:30.193070+00:00 node-115 kernel:
[87913.155256] [<ffffffff815c81b4>]
scsi_softirq_done+0x124/0x1502018-01-06T17:10:30.193071+00:00 node-115 kernel:
[87913.155258] [<ffffffff813d3787>]
blk_done_softirq+0x87/0xb02018-01-06T17:10:30.193087+00:00 node-115 kernel:
[87913.155263] [<ffffffff81085dc1>]
__do_softirq+0x101/0x2902018-01-06T17:10:30.193090+00:00 node-115 kernel:
[87913.155265] [<ffffffff810860c3>]
irq_exit+0xa3/0xb02018-01-06T17:10:30.193092+00:00 node-115 kernel:
[87913.155269] [<ffffffff81050e93>]
smp_call_function_single_interrupt+0x33/0x402018-01-06T17:10:30.193094+00:00
node-115 kernel: [87913.155270] [<ffffffff81845ae2>]
call_function_single_interrupt+0x82/0x902018-01-06T17:10:30.193095+00:00
node-115 kernel: [87913.155271] <EOI>2018-01-06T17:10:30.193096+00:00 node-115
kernel: [87913.155273] [<ffffffff81844414>] ?
_raw_spin_lock+0x14/0x302018-01-06T17:10:30.193098+00:00 node-115 kernel:
[87913.155275] [<ffffffff81842422>]
__mutex_lock_slowpath+0x72/0x1302018-01-06T17:10:30.193099+00:00 node-115
kernel: [87913.155289] [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120
[ocfs2]2018-01-06T17:10:30.193101+00:00 node-115 kernel: [87913.155291]
[<ffffffff818424ff>] mutex_lock+0x1f/0x302018-01-06T17:10:30.193102+00:00
node-115 kernel: [87913.155303] [<ffffffffc076177a>]
ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]2018-01-06T17:10:30.193104+00:00
node-115 kernel: [87913.155306] [<ffffffff812252c0>] ?
poll_select_copy_remaining+0x140/0x1402018-01-06T17:10:30.193105+00:00 node-115
kernel: [87913.155317] [<ffffffffc0760e20>] ?
ocfs2_check_range_for_refcount+0x150/0x150
[ocfs2]2018-01-06T17:10:30.193106+00:00 node-115 kernel: [87913.155321]
[<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d02018-01-06T17:10:30.193107+00:00
node-115 kernel: [87913.155324] [<ffffffff8122e8e5>] ?
__fget_light+0x25/0x602018-01-06T17:10:30.193108+00:00 node-115 kernel:
[87913.155325] [<ffffffff8122e933>] ?
__fdget+0x13/0x202018-01-06T17:10:30.193109+00:00 node-115 kernel:
[87913.155327] [<ffffffff812622cf>]
do_io_submit+0x25f/0x5002018-01-06T17:10:30.193109+00:00 node-115 kernel:
[87913.155329] [<ffffffff81262580>]
SyS_io_submit+0x10/0x202018-01-06T17:10:30.193110+00:00 node-115 kernel:
[87913.155331] [<ffffffff818446b2>]
entry_SYSCALL_64_fastpath+0x16/0x712018-01-06T17:10:30.193111+00:00 node-115
kernel: [87913.155332] Code: 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6
c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 8b 07
<84> c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f 1f 44
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
