Hi, I am having Juno multinode setup with 1Controller, 1 Network and 10 Compute nodes. There are 12 VMs running on that setup.
After some time one of my VM starts printing some strange messages on console repeatedly, which says: [78943.168505] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78946.592786] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78946.594413] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78947.617555] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78948.642004] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78949.600762] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78950.689530] Dead loop on virtual device vnf_sig_sp, fix it urgently! vnf_sig_sp is one interface created internally on that VM (not by neutron). After this VM runs in Kernel panic and then VM goes for a cyclic reboot. I am attaching the logs as well. Can anyone please suggest what should be done to fix this. Thanks in advance! BR, Varun
[78943.168505] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78946.592786] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78946.594413] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78947.617555] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78948.642004] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78949.600762] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78950.689530] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78952.290076] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78954.660765] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78955.592326] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78956.920754] Dead loop on virtual device vnf_sig_sp, fix it urgently! [78984.036141] Stack: [78984.036846] Call Trace: [78984.037802] Code: 19 e1 df ff 5b c3 0f 1f 80 00 00 00 00 53 48 89 fb e8 77 34 c0 ff b8 00 00 01 00 f0 0f c1 03 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 53 48 89 [78984.040002] Kernel panic - not syncing: softlockup: hung tasks [78984.040002] Pid: 1342, comm: java Tainted: PF ENX 3.0.101-0.47.52-default #1 [78984.040002] Call Trace: [78984.040002] [<ffffffff81004b95>] dump_trace+0x75/0x300 [78984.040002] [<ffffffff81461d43>] dump_stack+0x69/0x6f [78984.040002] [<ffffffff81461ddc>] panic+0x93/0x201 [78984.040002] [<ffffffff810c8261>] watchdog_timer_fn+0x181/0x190 [78984.040002] [<ffffffff8108842e>] __run_hrtimer+0xbe/0x1a0 [78984.040002] [<ffffffff81088739>] hrtimer_interrupt+0xd9/0x210 [78984.040002] [<ffffffff81026f43>] smp_apic_timer_interrupt+0x63/0xa0 [78984.040002] [<ffffffff8146d573>] apic_timer_interrupt+0x13/0x20 [78984.040002] [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30 [78984.060008] BUG: soft lockup - CPU#1 stuck for 22s! [java:1345] [78984.060009] Modules linked in: multimmap(PEN) tcp_diag inet_diag sch_htb xt_DROPDST(EN) ip_set_rule_ipportipportproto(EN) xt_set(E) ip_set_hash_ip(E) ip_set(E) ip_vs_rr(EN) ip6t_REJECT ipt_REJECT af_packet xt_tcpudp evip_sctpd(PFEN) nf_conntrack_ipv6(E) nf_defrag_ipv6 xfrm_user xt_multiport xt_connmark xt_state xt_mark xt_conntrack ip6table_mangle ip6table_filter iptable_filter ip_vs(EN) crc32c libcrc32c nf_conntrack_netlink nfnetlink ip6_tunnel(E) tunnel6 macvlan ev_drv(PEN) nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_HMARK(EN) ip6_tables(E) iptable_mangle ip_tables x_tables binfmt_misc mperf tipc(EX) softdog fuse loop dm_mod joydev usbhid hid sg sr_mod cdrom ata_generic ata_piix libata ipv6 ipv6_lib uhci_hcd rtc_cmos ehci_hcd xen_platform_pci acpiphp pci_hotplug processor thermal_sys hwmon serio_raw pcspkr scsi_mod usbcore button i2c_piix4 i2c_core usb_common nfs lockd fscache auth_rpcgss nfs_acl sunrpc virtio_balloon virtio_blk virtio_net virtio_pci virtio_ring virtio intel_agp intel_gtt floppy [last unloaded: multimmap] [78984.060044] Supported: No, Proprietary and Unsupported modules are loaded [78984.060045] CPU 1 [78984.060046] Modules linked in: multimmap(PEN) tcp_diag inet_diag sch_htb xt_DROPDST(EN) ip_set_rule_ipportipportproto(EN) xt_set(E) ip_set_hash_ip(E) ip_set(E) ip_vs_rr(EN) ip6t_REJECT ipt_REJECT af_packet xt_tcpudp evip_sctpd(PFEN) nf_conntrack_ipv6(E) nf_defrag_ipv6 xfrm_user xt_multiport xt_connmark xt_state xt_mark xt_conntrack ip6table_mangle ip6table_filter iptable_filter ip_vs(EN) crc32c libcrc32c nf_conntrack_netlink nfnetlink ip6_tunnel(E) tunnel6 macvlan ev_drv(PEN) nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_HMARK(EN) ip6_tables(E) iptable_mangle ip_tables x_tables binfmt_misc mperf tipc(EX) softdog fuse loop dm_mod joydev usbhid hid sg sr_mod cdrom ata_generic ata_piix libata ipv6 ipv6_lib uhci_hcd rtc_cmos ehci_hcd xen_platform_pci acpiphp pci_hotplug processor thermal_sys hwmon serio_raw pcspkr scsi_mod usbcore button i2c_piix4 i2c_core usb_common nfs lockd fscache auth_rpcgss nfs_acl sunrpc virtio_balloon virtio_blk virtio_net virtio_pci virtio_ring virtio intel_agp intel_gtt floppy [last unloaded: multimmap] [78984.060072] Supported: No, Proprietary and Unsupported modules are loaded [78984.060073] [78984.060074] Pid: 1345, comm: java Tainted: PF ENX 3.0.101-0.47.52-default #1 Fedora Project OpenStack Nova [78984.060077] RIP: 0010:[<ffffffff81464e6e>] [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30 [78984.060081] RSP: 0018:ffff88033999f7e8 EFLAGS: 00000297 [78984.060082] RAX: 000000000000cdcc RBX: dead000000200200 RCX: 0000000000001020 [78984.060083] RDX: 000000000000cdc6 RSI: ffffffff81a73580 RDI: ffffffffa0327b08 [78984.060085] RBP: 0000000000000206 R08: 0000000000000000 R09: 0000000000000002 [78984.060086] R10: ffff88033999f95c R11: ffff88033999fb38 R12: ffffffff8146d56e [78984.060087] R13: 00a800003999f81c R14: ffffffffa02f5da4 R15: ffff8804217f1240 [78984.060089] FS: 00007f1bf27ca700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000 [78984.060091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [78984.060092] CR2: 00007fca964bd000 CR3: 0000000391653000 CR4: 00000000001406e0 [78984.060097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [78984.060101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [78984.060102] Process java (pid: 1345, threadinfo ffff88033999e000, task ffff88033999c540) [78984.060103] Stack: [78984.060104] ffffffff81a73580 ffffffffa02f5dfc ffffffff81a73a00 ffffffff81a73a00 [78984.060106] ffffffff81a73580 ffffffffa02ef706 000000043999f8d8 ffff88033999fb4c [78984.060108] ffffffff81a73a00 ffff880422106000 0000000000000000 0000000000000000 [78984.060111] Call Trace: [78984.060122] [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib] [78984.060141] [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib] [78984.060151] [<ffffffff813aeed3>] dst_alloc+0x173/0x190 [78984.060159] [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib] [78984.060174] [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib] [78984.060189] [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib] [78984.060206] [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib] [78984.060246] [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv] [78984.060253] [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150 [78984.060253] [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib] [78984.060253] [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib] [78984.060253] [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib] [78984.060253] [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib] [78984.060253] [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710 [78984.060253] [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510 [78984.060253] [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60 [78984.060253] [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00 [78984.060253] [<ffffffff8138fce4>] sock_aio_write+0x114/0x130 [78984.060253] [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110 [78984.060253] [<ffffffff8115da5c>] vfs_write+0x12c/0x140 [78984.060253] [<ffffffff8115db73>] sys_write+0x53/0xa0 [78984.060253] [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b [78984.060253] [<00007f1c2e2a979d>] 0x7f1c2e2a979c [78984.060253] Code: 19 e1 df ff 5b c3 0f 1f 80 00 00 00 00 53 48 89 fb e8 77 34 c0 ff b8 00 00 01 00 f0 0f c1 03 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 53 48 89 [78984.060253] Call Trace: [78984.060253] [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib] [78984.060253] [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib] [78984.060253] [<ffffffff813aeed3>] dst_alloc+0x173/0x190 [78984.060253] [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib] [78984.060253] [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib] [78984.060253] [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib] [78984.060253] [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib] [78984.060253] [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv] [78984.060253] [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150 [78984.060253] [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib] [78984.060253] [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib] [78984.060253] [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib] [78984.060253] [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib] [78984.060253] [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710 [78984.060253] [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510 [78984.060253] [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60 [78984.060253] [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00 [78984.060253] [<ffffffff8138fce4>] sock_aio_write+0x114/0x130 [78984.060253] [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110 [78984.060253] [<ffffffff8115da5c>] vfs_write+0x12c/0x140 [78984.060253] [<ffffffff8115db73>] sys_write+0x53/0xa0 [78984.060253] [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b [78984.060253] [<00007f1c2e2a979d>] 0x7f1c2e2a979c [78984.060253] Kernel panic - not syncing: softlockup: hung tasks [78984.060253] Pid: 1345, comm: java Tainted: PF ENX 3.0.101-0.47.52-default #1 [78984.060253] Call Trace: [78984.060253] [<ffffffff81004b95>] dump_trace+0x75/0x300 [78984.060253] [<ffffffff81461d43>] dump_stack+0x69/0x6f [78984.060253] [<ffffffff81461ddc>] panic+0x93/0x201 [78984.060253] [<ffffffff810c8261>] watchdog_timer_fn+0x181/0x190 [78984.060253] [<ffffffff8108842e>] __run_hrtimer+0xbe/0x1a0 [78984.060253] [<ffffffff81088739>] hrtimer_interrupt+0xd9/0x210 [78984.060253] [<ffffffff81026f43>] smp_apic_timer_interrupt+0x63/0xa0 [78984.060253] [<ffffffff8146d573>] apic_timer_interrupt+0x13/0x20 [78984.060253] [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30 [78984.060253] [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib] [78984.060253] [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib] [78984.060253] [<ffffffff813aeed3>] dst_alloc+0x173/0x190 [78984.060253] [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib] [78984.060253] [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib] [78984.060253] [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib] [78984.060253] [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib] [78984.060253] [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv] [78984.060253] [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150 [78984.060253] [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib] [78984.060253] [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib] [78984.060253] [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib] [78984.060253] [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib] [78984.060253] [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710 [78984.060253] [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510 [78984.060253] [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60 [78984.060253] [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00 [78984.060253] [<ffffffff8138fce4>] sock_aio_write+0x114/0x130 [78984.060253] [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110 [78984.060253] [<ffffffff8115da5c>] vfs_write+0x12c/0x140 [78984.060253] [<ffffffff8115db73>] sys_write+0x53/0xa0 [78984.060253] [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b [78984.060253] [<00007f1c2e2a979d>] 0x7f1c2e2a979c [78984.040002] [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib] [78984.040002] [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib] [78984.040002] [<ffffffff813aeed3>] dst_alloc+0x173/0x190 [78984.040002] [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib] [78984.040002] [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib] [78984.040002] [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib] [78984.040002] [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib] [78984.040002] [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv] [78984.040002] [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150 [78984.040002] [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib] [78984.040002] [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib] [78984.040002] [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib] [78984.040002] [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib] [78984.040002] [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710 [78984.040002] [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510 [78984.040002] [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60 [78984.040002] [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00 [78984.040002] [<ffffffff8138fce4>] sock_aio_write+0x114/0x130 [78984.040002] [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110 [78984.040002] [<ffffffff8115da5c>] vfs_write+0x12c/0x140 [78984.040002] [<ffffffff8115db73>] sys_write+0x53/0xa0 [78984.040002] [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b [78984.040002] [<00007f1c2e2a979d>] 0x7f1c2e2a979c [78984.060253] Rebooting in 5 seconds.. [78984.040002] Rebooting in 5 seconds..
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
