Package: src:linux Version: 4.9.88-1 Hi,
I'm observing the attached errors on machines that are Xen dom0 and running the latest Debian Stretch 4.9 kernel as dom0 kernel. The errors have been happening a few times in the last few weeks. It started after upgrading them from Jessie and 3.16 kernel to Stretch with 4.9 kernel. For networking between domUs and the outside world, we use openvswitch. After such an error happens: * The amount of "flows" in the kernel quickly raises to the limit, 10000, as seen in output of ovs-dpctl show. * Network traffic that should flow through the openvswitch bridge starts disappearing in a seemingly random way. * The memory usage of the userspace ovs-vswitchd starts growing quickly. * Many of the ovs commands, like to add or remove an interface or bridge hang. After a restart of the openvswitch-switch service, and fixing up a bunch of configuration of connected interfaces, functionality is restored. While most of the symptoms seem related to userspace openvswitch processes, the cause of it all seems to be in the kernel, while the userspace ovs-vswitchd process is receiving a network packet? Sadly I do not know how to reproduce this, except for just waiting until it happens again. Please advice what else I could use to help resolving this issue. Thanks, Regards, -- Hans van Kranenburg
May 4 08:23:03 altair kernel: [83978.662075] BUG: unable to handle kernel paging request at 000000030000001f May 4 08:23:03 altair kernel: [83978.665887] IP: [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110 May 4 08:23:03 altair kernel: [83978.669837] PGD 0 May 4 08:23:03 altair kernel: [83978.669882] May 4 08:23:03 altair kernel: [83978.673589] Oops: 0000 [#1] SMP May 4 08:23:03 altair kernel: [83978.677281] Modules linked in: cls_u32 sch_ingress act_mirred sch_fq_codel ifb xt_mark sch_htb xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time binfmt_misc xen_gntdev xen_evtchn openvswitch nf_nat_ipv6 libcrc32c xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt intel_powerclamp crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel pcspkr serio_raw joydev evdev amdkfd radeon ttm drm_kms_helper drm i2c_algo_bit lpc_ich mfd_core i7core_edac hpilo May 4 08:23:03 altair kernel: [83978.701936] sg ipmi_si hpwdt edac_core ipmi_msghandler acpi_power_meter button shpchp dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ehci_pci uhci_hcd ehci_hcd usbcore usb_common hpsa scsi_transport_sas bnx2 mlx4_core devlink scsi_mod thermal May 4 08:23:03 altair kernel: [83978.724406] CPU: 1 PID: 1486 Comm: revalidator7 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.88-1 May 4 08:23:03 altair kernel: [83978.729139] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015 May 4 08:23:03 altair kernel: [83978.733958] task: ffff880119e1ee80 task.stack: ffffc90042764000 May 4 08:23:03 altair kernel: [83978.738724] RIP: e030:[<ffffffff814f5c7d>] [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110 May 4 08:23:03 altair kernel: [83978.743560] RSP: e02b:ffffc90042767c78 EFLAGS: 00010206 May 4 08:23:03 altair kernel: [83978.748352] RAX: 0000000000000050 RBX: 00000002ffffffff RCX: ffffffff81ce0f40 May 4 08:23:03 altair kernel: [83978.753116] RDX: ffffffffffffffff RSI: ffff8800cc998900 RDI: ffff8800cc998900 May 4 08:23:03 altair kernel: [83978.757867] RBP: ffff8800cc998900 R08: ffff880123c00000 R09: ffff88011f220000 May 4 08:23:03 altair kernel: [83978.762598] R10: ffff8800cc998900 R11: ffff880119e10280 R12: 0000000000000002 May 4 08:23:03 altair kernel: [83978.767321] R13: ffff88011f227ec0 R14: ffff88011dea2800 R15: 0000000000000000 May 4 08:23:03 altair kernel: [83978.772000] FS: 00007fc1656cc700(0000) GS:ffff880128240000(0000) knlGS:0000000000000000 May 4 08:23:03 altair kernel: [83978.776671] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 May 4 08:23:03 altair kernel: [83978.781355] CR2: 000000030000001f CR3: 00000001212b1000 CR4: 0000000000002660 May 4 08:23:03 altair kernel: [83978.786135] Stack: May 4 08:23:03 altair kernel: [83978.790841] ffff880120a28000 ffff8800cc998900 ffffc90042767ec0 0000000000007ea4 May 4 08:23:03 altair kernel: [83978.795898] ffffffff814f6267 ffff880120a28000 ffff8800cc998900 ffffffff814fcc91 May 4 08:23:03 altair kernel: [83978.800806] ffff880120a28000 ffffffff8153f2df ffffc90000000000 ffff8800cc998900 May 4 08:23:03 altair kernel: [83978.805723] Call Trace: May 4 08:23:03 altair kernel: [83978.810654] [<ffffffff814f6267>] ? consume_skb+0x27/0x80 May 4 08:23:03 altair kernel: [83978.815626] [<ffffffff814fcc91>] ? skb_free_datagram+0x11/0x40 May 4 08:23:03 altair kernel: [83978.820545] [<ffffffff8153f2df>] ? netlink_recvmsg+0x19f/0x440 May 4 08:23:03 altair kernel: [83978.825426] [<ffffffff814ed4ca>] ? ___sys_recvmsg+0xda/0x1f0 May 4 08:23:03 altair kernel: [83978.830273] [<ffffffff812237fb>] ? file_update_time+0xcb/0x110 May 4 08:23:03 altair kernel: [83978.835058] [<ffffffff8120fbeb>] ? pipe_write+0x29b/0x3e0 May 4 08:23:03 altair kernel: [83978.839800] [<ffffffff812066b0>] ? new_sync_write+0xe0/0x130 May 4 08:23:03 altair kernel: [83978.844502] [<ffffffff814edf4e>] ? __sys_recvmsg+0x4e/0x90 May 4 08:23:03 altair kernel: [83978.849161] [<ffffffff81003b7d>] ? do_syscall_64+0x8d/0xf0 May 4 08:23:03 altair kernel: [83978.853779] [<ffffffff8161244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6 May 4 08:23:03 altair kernel: [83978.858397] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48 May 4 08:23:03 altair kernel: [83978.868227] RIP [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110 May 4 08:23:03 altair kernel: [83978.873017] RSP <ffffc90042767c78> May 4 08:23:03 altair kernel: [83978.877746 May 4 22:00:22 sirius kernel: [1999361.378086] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0 May 4 22:00:22 sirius kernel: [1999361.381804] IP: [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110 May 4 22:00:22 sirius kernel: [1999361.385492] PGD 0 May 4 22:00:22 sirius kernel: [1999361.385535] May 4 22:00:22 sirius kernel: [1999361.389145] Oops: 0000 [#1] SMP May 4 22:00:22 sirius kernel: [1999361.392725] Modules linked in: xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time binfmt_misc openvswitch nf_nat_ipv6 libcrc32c xen_gntdev xen_evtchn xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt intel_powerclamp crct10dif_pclmul crc32_pclmul amdkfd iTCO_wdt evdev joydev iTCO_vendor_support ghash_clmulni_intel radeon ttm serio_raw pcspkr drm_kms_helper drm i2c_algo_bit sg i7core_edac lpc_ich ipmi_si acpi_power_meter hpilo hpwdt mfd_core edac_core ipmi_msghandler button May 4 22:00:22 sirius kernel: [1999361.416634] shpchp dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ehci_pci uhci_hcd ehci_hcd usbcore usb_common mlx4_core hpsa scsi_transport_sas bnx2 devlink scsi_mod thermal May 4 22:00:22 sirius kernel: [1999361.438322] CPU: 2 PID: 1400 Comm: revalidator9 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.82-1+deb9u3 May 4 22:00:22 sirius kernel: [1999361.442773] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015 May 4 22:00:22 sirius kernel: [1999361.447219] task: ffff880111c58540 task.stack: ffffc90041bcc000 May 4 22:00:22 sirius kernel: [1999361.451796] RIP: e030:[<ffffffff814f4c7d>] [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110 May 4 22:00:22 sirius kernel: [1999361.456294] RSP: e02b:ffffc90041bcfc78 EFLAGS: 00010206 May 4 22:00:22 sirius kernel: [1999361.460758] RAX: 0000000000000030 RBX: 00000000000001c0 RCX: ffffffff81ce0e00 May 4 22:00:22 sirius kernel: [1999361.465261] RDX: 0000000000008100 RSI: ffff880118a94f00 RDI: ffff880118a94f00 May 4 22:00:22 sirius kernel: [1999361.469724] RBP: ffff880118a94f00 R08: ffff88011bc00000 R09: ffff8800b0218000 May 4 22:00:22 sirius kernel: [1999361.474230] R10: ffff880118a94f00 R11: ffff880111c50240 R12: 0000000000000000 May 4 22:00:22 sirius kernel: [1999361.478710] R13: ffff8800b021fec0 R14: ffff8800b8356a40 R15: 0000000000000000 May 4 22:00:22 sirius kernel: [1999361.483220] FS: 00007faa54946700(0000) GS:ffff880120280000(0000) knlGS:0000000000000000 May 4 22:00:22 sirius kernel: [1999361.487736] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 May 4 22:00:22 sirius kernel: [1999361.492181] CR2: 00000000000001e0 CR3: 0000000119ce9000 CR4: 0000000000002660 May 4 22:00:22 sirius kernel: [1999361.496661] Stack: May 4 22:00:22 sirius kernel: [1999361.501036] ffff8801190a2800 ffff880118a94f00 ffffc90041bcfec0 0000000000007eac May 4 22:00:22 sirius kernel: [1999361.505470] ffffffff814f5267 ffff8801190a2800 ffff880118a94f00 ffffffff814fbc91 May 4 22:00:22 sirius kernel: [1999361.509844] ffff8801190a2800 ffffffff8153e2bf ffffc90000000000 ffff880118a94f00 May 4 22:00:22 sirius kernel: [1999361.514213] Call Trace: May 4 22:00:22 sirius kernel: [1999361.518499] [<ffffffff814f5267>] ? consume_skb+0x27/0x80 May 4 22:00:22 sirius kernel: [1999361.522818] [<ffffffff814fbc91>] ? skb_free_datagram+0x11/0x40 May 4 22:00:22 sirius kernel: [1999361.527109] [<ffffffff8153e2bf>] ? netlink_recvmsg+0x19f/0x440 May 4 22:00:22 sirius kernel: [1999361.531314] [<ffffffff814ec4ca>] ? ___sys_recvmsg+0xda/0x1f0 May 4 22:00:22 sirius kernel: [1999361.535488] [<ffffffff812221ab>] ? file_update_time+0xcb/0x110 May 4 22:00:22 sirius kernel: [1999361.539626] [<ffffffff8120e5cb>] ? pipe_write+0x29b/0x3e0 May 4 22:00:22 sirius kernel: [1999361.543790] [<ffffffff812050a0>] ? new_sync_write+0xe0/0x130 May 4 22:00:22 sirius kernel: [1999361.547989] [<ffffffff814ecf4e>] ? __sys_recvmsg+0x4e/0x90 May 4 22:00:22 sirius kernel: [1999361.552218] [<ffffffff81003b7f>] ? do_syscall_64+0x8f/0xf0 May 4 22:00:22 sirius kernel: [1999361.556467] [<ffffffff816113b8>] ? entry_SYSCALL_64_after_swapgs+0x42/0xb0 May 4 22:00:22 sirius kernel: [1999361.560791] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48 May 4 22:00:22 sirius kernel: [1999361.570202] RIP [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110 May 4 22:00:22 sirius kernel: [1999361.575033] RSP <ffffc90041bcfc78> May 4 22:00:22 sirius kernel: [1999361.579731] CR2: 00000000000001e0 May 4 22:00:22 sirius kernel: [1999361.599233] ---[ end trace de6345fc470c5362 ]--- May 18 13:49:26 omega kernel: [1213243.942643] general protection fault: 0000 [#1] SMP May 18 13:49:26 omega kernel: [1213243.946704] Modules linked in: xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time xen_gntdev openvswitch xen_evtchn nf_nat_ipv6 libcrc32c xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt amdkfd radeon intel_powerclamp crct10dif_pclmul iTCO_wdt crc32_pclmul iTCO_vendor_support ttm ghash_clmulni_intel hpwdt pcspkr drm_kms_helper drm serio_raw evdev i2c_algo_bit joydev sg hpilo lpc_ich mfd_core i7core_edac ipmi_si edac_core ipmi_msghandler acpi_power_meter shpchp button dm_multipath May 18 13:49:26 omega kernel: [1213243.973478] dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse uhci_hcd ehci_pci ehci_hcd usbcore usb_common hpsa bnx2 mlx4_core scsi_transport_sas devlink scsi_mod thermal May 18 13:49:26 omega kernel: [1213243.997290] CPU: 2 PID: 1582 Comm: revalidator9 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.88-1 May 18 13:49:26 omega kernel: [1213244.002200] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015 May 18 13:49:26 omega kernel: [1213244.007157] task: ffff8801186caf00 task.stack: ffffc90041b8c000 May 18 13:49:26 omega kernel: [1213244.012040] RIP: e030:[<ffffffff814f5c7d>] [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110 May 18 13:49:26 omega kernel: [1213244.016957] RSP: e02b:ffffc90041b8fc78 EFLAGS: 00010206 May 18 13:49:26 omega kernel: [1213244.021783] RAX: 0000000000000030 RBX: 290008a753b675a9 RCX: ffffffff81ce0f40 May 18 13:49:26 omega kernel: [1213244.026673] RDX: 0000000000008100 RSI: ffff88011a2c6200 RDI: ffff88011a2c6200 May 18 13:49:26 omega kernel: [1213244.031596] RBP: ffff88011a2c6200 R08: ffff88011bc00000 R09: ffff88011aa70000 May 18 13:49:26 omega kernel: [1213244.036422] R10: ffff88011a2c6200 R11: ffff8801186c0200 R12: 0000000000000000 May 18 13:49:26 omega kernel: [1213244.041267] R13: ffff88011aa77ec0 R14: ffff8801199da7c0 R15: 0000000000000000 May 18 13:49:26 omega kernel: [1213244.046055] FS: 00007fe5f35e2700(0000) GS:ffff880120280000(0000) knlGS:0000000000000000 May 18 13:49:26 omega kernel: [1213244.050785] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 May 18 13:49:26 omega kernel: [1213244.055488] CR2: 00007fe579f4f059 CR3: 0000000117428000 CR4: 0000000000002660 May 18 13:49:26 omega kernel: [1213244.060232] Stack: May 18 13:49:26 omega kernel: [1213244.064896] ffff880117588800 ffff88011a2c6200 ffffc90041b8fec0 0000000000007e94 May 18 13:49:26 omega kernel: [1213244.069725] ffffffff814f6267 ffff880117588800 ffff88011a2c6200 ffffffff814fcc91 May 18 13:49:26 omega kernel: [1213244.074552] ffff880117588800 ffffffff8153f2df ffffc90000000000 ffff88011a2c6200 May 18 13:49:26 omega kernel: [1213244.079377] Call Trace: May 18 13:49:26 omega kernel: [1213244.084123] [<ffffffff814f6267>] ? consume_skb+0x27/0x80 May 18 13:49:26 omega kernel: [1213244.089047] [<ffffffff814fcc91>] ? skb_free_datagram+0x11/0x40 May 18 13:49:26 omega kernel: [1213244.093728] [<ffffffff8153f2df>] ? netlink_recvmsg+0x19f/0x440 May 18 13:49:26 omega kernel: [1213244.098359] [<ffffffff814ed4ca>] ? ___sys_recvmsg+0xda/0x1f0 May 18 13:49:26 omega kernel: [1213244.102962] [<ffffffff812237fb>] ? file_update_time+0xcb/0x110 May 18 13:49:26 omega kernel: [1213244.107530] [<ffffffff8120fbeb>] ? pipe_write+0x29b/0x3e0 May 18 13:49:26 omega kernel: [1213244.112074] [<ffffffff812066b0>] ? new_sync_write+0xe0/0x130 May 18 13:49:26 omega kernel: [1213244.116625] [<ffffffff814edf4e>] ? __sys_recvmsg+0x4e/0x90 May 18 13:49:26 omega kernel: [1213244.121183] [<ffffffff81003b7d>] ? do_syscall_64+0x8d/0xf0 May 18 13:49:26 omega kernel: [1213244.125715] [<ffffffff8161244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6 May 18 13:49:26 omega kernel: [1213244.130196] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48 May 18 13:49:26 omega kernel: [1213244.139830] RIP [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110 May 18 13:49:26 omega kernel: [1213244.144491] RSP <ffffc90041b8fc78> May 18 13:49:26 omega kernel: [1213244.164037] ---[ end trace c53e06696e145c33 ]---
