I have a couple of 2-node clusters running a bunch of KVM guests, they run
DRBD active/active with OCFS2 as a cluster filesystem.
All the hosts run Gentoo Hardened.
I recently updated one of the hosts in the "test" cluster to kernel 4.1.6,
first to the Gentoo Hardened sources, and when that crashed, I've just tried
the equivalent gentoo-sources 4.1.6.
The panic trace seems to point to OCFS2 - both kernels crash immediately as
soon as a single KVM guest starts to mount its root, with the following:
Sep 10 09:59:05 hades kernel: ------------[ cut here ]------------
Sep 10 09:59:05 hades kernel: kernel BUG at fs/ocfs2/dlmglue.c:775!
Sep 10 09:59:05 hades kernel: invalid opcode: 0000 [#1] SMP
Sep 10 09:59:05 hades kernel: Modules linked in: vhost_net vhost macvtap
macvlan tun drbd lru_cache ip6t_REJECT nf_reject_ipv6 ipt_REJECT
nf_reject_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack xt_tcpudp bridge
ip6table_filter 8021q garp stp mrp llc ip6_tables iptable_filter
nf_conntrack_ftp nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables
x_tables osst ch st ipv6 dm_zero dm_thin_pool dm_persistent_data dm_bio_prison
dm_round_robin dm_multipath scsi_dh virtio_pci virtio_balloon virtio_ring
virtio xts gf128mul aes_x86_64 cbc sha512_generic sha256_generic sha1_generic
scsi_transport_iscsi nfs lockd grace sunrpc multipath linear raid10 raid1
raid0 dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor
async_tx raid6_pq dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log
dm_mod hid_sunplus
Sep 10 09:59:05 hades kernel: hid_sony led_class hid_samsung hid_pl
hid_petalynx hid_gyration sl811_hcd xhci_pci xhci_hcd ohci_pci ohci_hcd
usb_storage megaraid_sas megaraid_mbox megaraid_mm mptsas mptfc
scsi_transport_fc mptspi scsi_transport_spi mptscsih mptbase sg pdc_adma
sata_inic162x sata_mv ahci libahci sata_qstor sata_vsc sata_uli sata_sis
sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise
pata_sl82c105 pata_via pata_marvell pata_sis pata_netcell pata_pdc202xx_old
pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415
pata_ns87410 pata_serverworks pata_artop pata_it821x pata_optidma pata_hpt3x2n
pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000
pata_sil680 pata_radisys pata_pdc2027x pata_mpiix joydev usbhid uhci_hcd
coretemp kvm_intel kvm crc32c_intel
Sep 10 09:59:05 hades kernel: microcode pcspkr pata_acpi ehci_pci ehci_hcd
ixgbe ata_piix pata_jmicron arcmsr mdio i2c_i801 libata igb usbcore mpt2sas
ptp usb_common raid_class pps_core i2c_algo_bit scsi_transport_sas ioatdma
i2c_core dca button acpi_cpufreq processor thermal_sys
Sep 10 09:59:05 hades kernel: CPU: 0 PID: 5363 Comm: drbd_a_drbd0 Tainted: G
I 4.1.6-gentoo #1
Sep 10 09:59:05 hades kernel: Hardware name: Supermicro X8DAH/X8DAH, BIOS 2.0
06/01/2010
Sep 10 09:59:05 hades kernel: task: ffff88180cfe8050 ti: ffff88180bb58000
task.ti:
ffff88180bb58000
Sep 10 09:59:05 hades kernel: RIP: 0010:[<ffffffff81227ab7>]
[<ffffffff81227ab7>]
__ocfs2_cluster_unlock.isra.43+0x40/0x9e
Sep 10 09:59:05 hades kernel: RSP: 0018:ffff88180bb5bb88 EFLAGS: 00010046
Sep 10 09:59:05 hades kernel: RAX: 0000000000000000 RBX: ffff88180ef3a108 RCX:
00000000000000a4
Sep 10 09:59:05 hades kernel: RDX: 000000000000a4a4 RSI: ffff88180ef3a108 RDI:
ffff88180ef3a174
Sep 10 09:59:05 hades kernel: RBP: ffff88180bb5bbb8 R08: ffffffff81211f6f R09:
0000000000000001
Sep 10 09:59:05 hades kernel: R10: 0000000000000000 R11: 000000000000d2e0 R12:
ffff88180ef3a174
Sep 10 09:59:05 hades kernel: R13: 0000000000000005 R14: ffff88180c6d5000 R15:
0000000000000246
Sep 10 09:59:05 hades kernel: FS: 0000000000000000(0000)
GS:ffff880c3fc00000(0000) knlGS:0000000000000000
Sep 10 09:59:05 hades kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Sep 10 09:59:05 hades kernel: CR2: 0000000000509b2f CR3: 000000000161f000 CR4:
00000000000026f0
Sep 10 09:59:05 hades kernel: Stack:
Sep 10 09:59:05 hades kernel: 0000000000000000 0000000000000005
ffff88180ef3a508 ffff88180c6d5000
Sep 10 09:59:05 hades kernel: 0000000af8310400 0000000000000001
ffff88180bb5bbf8 ffffffff81229141
Sep 10 09:59:05 hades kernel: 000000000000000a ffff880c0ea43f40
ffff880c0eedeb40
ffff880c035cf040
Sep 10 09:59:05 hades kernel: Call Trace:
Sep 10 09:59:05 hades kernel: [<ffffffff81229141>] ocfs2_rw_unlock+0xbc/0xc7
Sep 10 09:59:05 hades kernel: [<ffffffff81211fc8>] ocfs2_dio_end_io+0x59/0x5e
Sep 10 09:59:05 hades kernel: [<ffffffff81113bc4>] dio_complete+0x92/0x150
Sep 10 09:59:05 hades kernel: [<ffffffff81113d43>] dio_bio_end_aio+0xc1/0xca
Sep 10 09:59:05 hades kernel: [<ffffffff812c0bc5>] bio_endio+0x61/0x68
Sep 10 09:59:05 hades kernel: [<ffffffffa1538ef4>]
complete_master_bio+0x1f/0x145
[drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa1533368>]
validate_req_change_req_state+0xca/0xdb [drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa1533623>] got_BlockAck+0x113/0x130
[drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa1537ff9>] drbd_asender+0x58b/0x6c3
[drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa153ed22>] ?
drbd_destroy_connection+0xaf/0xaf [drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa153ed68>]
drbd_thread_setup+0x46/0x114
[drbd]
Sep 10 09:59:05 hades kernel: [<ffffffffa153ed22>] ?
drbd_destroy_connection+0xaf/0xaf [drbd]
Sep 10 09:59:05 hades kernel: [<ffffffff81050497>] kthread+0xcd/0xd5
Sep 10 09:59:05 hades kernel: [<ffffffff810503ca>] ?
kthread_create_on_node+0x16c/0x16c
Sep 10 09:59:05 hades kernel: [<ffffffff81490b92>] ret_from_fork+0x42/0x70
Sep 10 09:59:05 hades kernel: [<ffffffff810503ca>] ?
kthread_create_on_node+0x16c/0x16c
Sep 10 09:59:05 hades kernel: Code: 6c 53 4c 89 e7 48 89 f3 51 e8 9e 89 26 00
48 85 db 75 02 0f 0b 41 83 fd 03 49 89 c7 74 16 41 83 fd 05 75 20 8b 43 5c 85
c0 75 02 <0f> 0b ff c8 89 43 5c eb 12 8b 53 58 85 d2 75 02 0f 0b ff ca 89
Sep 10 09:59:05 hades kernel: RIP [<ffffffff81227ab7>]
__ocfs2_cluster_unlock.isra.43+0x40/0x9e
Sep 10 09:59:05 hades kernel: RSP <ffff88180bb5bb88>
Sep 10 09:59:05 hades kernel: ---[ end trace 42d7ee8da6efb352 ]---
These hosts have all run 3.18.9 for the last 5 months with no issues, and
previous 3.x kernels also with no problems since installation about a year
ago.
If anyone has a clue what I'm doing wrong, I'd love to hear from you ...
thanks for any help.
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
