Okay. On a different machine with different load, we've now got the same
problem:
KERNEL: /usr/lib/debug/boot/vmlinux-4.4.0-78-generic
DUMPFILE: dump.201705300948 [PARTIAL DUMP]
CPUS: 4
DATE: Tue May 30 09:48:05 2017
UPTIME: 01:47:38
LOAD AVERAGE: 0.05, 0.06, 0.04
TASKS: 292
NODENAME: ossoio-docker1-tcn
RELEASE: 4.4.0-78-generic
VERSION: #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017
MACHINE: x86_64 (2199 Mhz)
MEMORY: 4 GB
PANIC: "BUG: unable to handle kernel paging request at ffff88013a404000"
PID: 0
COMMAND: "swapper/3"
TASK: ffff88013abf1980 (1 of 4) [THREAD_INFO: ffff88013a400000]
CPU: 3
STATE: TASK_RUNNING (PANIC)
#9 [ffff88013a403e20] async_page_fault at ffffffff81842be8
#10 [ffff88013a403e38] tick_nohz_idle_exit at ffffffff810ff75e
#11 [ffff88013a403ed8] cpu_startup_entry at ffffffff810c4736
#12 [ffff88013a403f30] start_secondary at ffffffff810517c4
Differences:
- this machine does not use zfs, the other one does
- this machine runs docker instances, the other one mainly mysqld
- this machines has x2apic enabled according to dmesg (no idea what that is)
Similarities:
- both are KVM guests (same KVM cluster, different nodes)
- the two KVM nodes have the same hardware, same kernel and same KVM host
software
- 4 cpus, 2200MHz, flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl
xtopology pni cx16 x2apic hypervisor lahf_lm
- 512MB swap
Modules loaded:
- both: 8250_fintek autofs4 drm drm_kms_helper fb_sys_fops floppy
i2c_piix4 input_leds mac_hid parport parport_pc pata_acpi ppdev psmouse
serio_raw shpchp syscopyarea sysfillrect sysimgblt ttm
- this: aufs bridge br_netfilter ip6table_filter ip6_tables
iptable_filter iptable_nat ip_tables ipt_MASQUERADE llc nf_conntrack
nf_conntrack_ipv4 nf_conntrack_netlink nf_defrag_ipv4 nf_nat nf_nat_ipv4
nf_nat_masquerade_ipv4 nfnetlink stp veth xfrm_algo xfrm_user x_tables
xt_addrtype xt_conntrack xt_nat xt_tcpudp
- other: spl(O) zavl(PO) zcommon(PO) zfs(PO) znvpair(PO) zunicode(PO)
We should be able to run this node on the vanilla kernel and see how that goes.
Will report back in a bit.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1691741
Title:
Execute NX-protected page - 4.4.0-78-generic - kernel panic
Status in linux package in Ubuntu:
Confirmed
Bug description:
After upgrading from 4.4.0-77 to 4.4.0-78 I started getting kernel
panics.
The crashes do not happen immediately, but have happened generally
after a couple of minutes, sometimes more.
After enabling linux-crashdump stuff, I managed to extract this dmesg.
[ 995.103846] kernel tried to execute NX-protected page - exploit attempt?
(uid: 0)
[ 995.104141] BUG: unable to handle kernel paging request at ffff88042a284000
[ 995.104407] IP: [<ffff88042a284000>] 0xffff88042a284000
[ 995.104594] PGD 43f20b067 PUD 43f20e067 PMD 42a3da063 PTE 800000042a284163
[ 995.104946] Oops: 0011 [#1] SMP
[ 995.105143] Modules linked in: zfs(PO) zunicode(PO) zcommon(PO)
znvpair(PO) spl(O) zavl(PO) ppdev input_leds shpchp serio_raw i2c_piix4 mac_hid
parport_pc parport 8250_fintek autofs4 ttm drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops drm psmouse floppy pata_acpi
[ 995.107081] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P O
4.4.0-78-generic #99-Ubuntu
[ 995.107299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[ 995.107573] task: ffff88042a278000 ti: ffff88042a280000 task.ti:
ffff88042a280000
[ 995.108070] RIP: 0010:[<ffff88042a284000>] [<ffff88042a284000>]
0xffff88042a284000
[ 995.108637] RSP: 0018:ffff88042a283ed0 EFLAGS: 00010082
[ 995.109116] RAX: 0000000000000001 RBX: 000000e797438af0 RCX:
0000000000000000
[ 995.109638] RDX: 0000000000000001 RSI: 0000000000000083 RDI:
0000000000000083
[ 995.110143] RBP: ffffffff81f38d40 R08: 000000000000000a R09:
0000000000000000
[ 995.110665] R10: 000000010002a665 R11: 0000000000004c00 R12:
ffff88042a283ed0
[ 995.111182] R13: ffffffff810ff75e R14: 0000000000000000 R15:
ffff88042a280000
[ 995.111733] FS: 0000000000000000(0000) GS:ffff88043fc80000(0000)
knlGS:0000000000000000
[ 995.112486] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 995.112978] CR2: ffff88042a284000 CR3: 000000043d246000 CR4:
00000000000006e0
[ 995.113497] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 995.114085] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 995.114612] Stack:
[ 995.114965] ffff88042a283f28 ffffffff810c4736 ffff88042a280000
ffff88042a284000
[ 995.116204] ee041b0196f77cc4 a1abbcd2b8b123ce 0000000000000000
0000000000000000
[ 995.117389] 0000000000000000 0000000000000000 0000000000000000
ffff88042a283f48
[ 995.118425] Call Trace:
[ 995.118811] [<ffffffff810c4736>] ? cpu_startup_entry+0x176/0x350
[ 995.119293] [<ffffffff810517c4>] ? start_secondary+0x154/0x190
[ 995.119775] Code: ff ff ff 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00
02 02 00 00 00 00 00 00 58 3f 28 2a 04 88 ff ff 18 00 00 00 00 00 00 00 <c0> 8c
27 2a 04 88 ff ff 00 00 00 00 00 00 00 00 02 00 00 00 00
[ 995.125554] RIP [<ffff88042a284000>] 0xffff88042a284000
[ 995.126088] RSP <ffff88042a283ed0>
[ 995.126453] CR2: ffff88042a284000
I've upgraded other machines as well, and only this particular VM
shows this behaviour.
I have a crash dump, but I haven't looked into the contents yet.
Getting the dmesg was already a pain in the behind.
The VM this happens on is:
- a KVM guest
- x86_64, 4 cores
- 16gb ram
lsb_release:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
lspci says:
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton
II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: VMware SVGA II Adapter
00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:0a.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:0b.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:12.0 Ethernet controller: Red Hat, Inc Virtio network device
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
Let me know if there are other helpful details I can provide. If I
find out more, I'll update this ticket.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1691741/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
