Analysing core dump with crash tool: ---- SYSTEM MAP: /boot/System.map-3.2.0-80-virtual DEBUG KERNEL: /usr/lib/debug/boot/vmlinux-3.2.0-82-virtual DUMPFILE: ./VmCore CPUS: 2 DATE: Sat Jun 6 00:23:24 2015 UPTIME: 1 days, 11:37:29 LOAD AVERAGE: 1.96, 1.11, 0.54 TASKS: 183 NODENAME: activemq2 RELEASE: 3.2.0-80-virtual VERSION: #116-Ubuntu SMP Mon Mar 23 17:28:52 UTC 2015 MACHINE: x86_64 (1799 Mhz) MEMORY: 4 GB PANIC: "[128249.422864] Kernel panic - not syncing: softlockup: hung tasks" PID: 6345 COMMAND: "java" TASK: ffff880037412de0 [THREAD_INFO: ffff880037694000] CPU: 0 STATE: TASK_RUNNING (PANIC) ---- PID: 6345 TASK: ffff880037412de0 CPU: 0 COMMAND: "java" #0 [ffff88011fc03cd0] machine_kexec at ffffffff8103970a ... #7 [ffff88011fc03fb0] apic_timer_interrupt at ffffffff81667b9e --- <IRQ stack> --- #8 [ffff880037695cd8] apic_timer_interrupt at ffffffff81667b9e [exception RIP: flush_tlb_others_ipi+290] RIP: ffffffff81046932 RSP: ffff880037695d88 RFLAGS: 00000202 RAX: 0000000000000002 RBX: ffff880117c06a00 RCX: 0000000000000002 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000296 RBP: ffff880037695db8 R8: ffffffff81e0d0a0 R9: 0000000000000040 R10: 0000000000000000 R11: ffff8801163df4d0 R12: ffff880037695d28 R13: 0000000000000000 R14: ffffffff81143575 R15: ffff880037695d68 ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018 ----
It is clear that we are locked inside flush_tlb_others_ipi. Disassembling flush_tlb_others_ipi I could see we are locked inside a loop waiting for a cpumask to be emptied. This cpumask is emptied when all other CPUs received the IPI and processed the event "INVALIDATE_TLB_VECTOR_START" properly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1467955 Title: Precise BUG: soft lockup in flush_tlb_others_ipi Status in linux package in Ubuntu: Fix Released Status in linux source package in Precise: In Progress Bug description: The following stack trace (with kernel dump) was brought to me: """ [1796904.032010] BUG: soft lockup - CPU#0 stuck for 23s! [java:6383] [1796904.036004] Modules linked in: isofs psmouse virtio_balloon serio_raw acpiphp floppy [1796904.036004] CPU 0 [1796904.036004] Modules linked in: isofs psmouse virtio_balloon serio_raw acpiphp floppy [1796904.036004] [1796904.036004] Pid: 6383, comm: java Not tainted 3.2.0-76-virtual #111-Ubuntu OpenStack Foundation OpenStack Nova [1796904.036004] RIP: 0010:[<ffffffff81046922>] [<ffffffff81046922>] flush_tlb_others_ipi+0x122/0x130 [1796904.036004] RSP: 0018:ffff880065791d58 EFLAGS: 00000202 [1796904.036004] RAX: 0000000000000002 RBX: ffffea0003470bf0 RCX: 0000000000000002 [1796904.036004] RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000296 [1796904.036004] RBP: ffff880065791d88 R08: ffffffff81e0c0a0 R09: 0000000000000040 [1796904.036004] R10: ffffea0003471240 R11: 0000000000000000 R12: ffff880065791e20 [1796904.036004] R13: ffff880059e96f20 R14: ffff880116249848 R15: 00ff880065791d78 [1796904.036004] FS: 00007f83612d2700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000 [1796904.036004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1796904.036004] CR2: 00007f83be381420 CR3: 0000000118be0000 CR4: 00000000000006f0 [1796904.036004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1796904.036004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1796917.981999] Process java (pid: 6383, threadinfo ffff880065790000, task ffff880053c0dbc0) [1796917.981999] Stack: [1796917.981999] 00007f83612ccfff ffff880059e96f20 ffff880116200e00 ffff8801162010d0 [1796917.981999] 00007f83612cd000 ffff880116200e00 ffff880065791d98 ffffffff81046aae [1796917.981999] ffff880065791db8 ffffffff81046b7b 00007f83611d5000 ffff880065791e20 [1796917.981999] Call Trace: [1796917.982394] ata2: lost interrupt (Status 0x58) [1796917.981999] [<ffffffff81046aae>] native_flush_tlb_others+0xe/0x10 [1796917.981999] [<ffffffff81046b7b>] flush_tlb_mm+0x5b/0xa0 [1796917.981999] [<ffffffff8113ba06>] tlb_flush_mmu+0x46/0x90 [1796917.981999] [<ffffffff8113ba64>] tlb_finish_mmu+0x14/0x40 [1796917.981999] [<ffffffff8113e3a7>] zap_page_range+0xb7/0xd0 [1796917.981999] [<ffffffff8113a85d>] madvise_vma+0xfd/0x140 [1796917.981999] [<ffffffff8107b917>] ? __set_task_blocked+0x37/0x80 [1796917.981999] [<ffffffff81095b27>] ? getnstimeofday+0x57/0xe0 [1796917.981999] [<ffffffff8113aa7e>] sys_madvise+0x1de/0x280 [1796917.981999] [<ffffffff81666b82>] system_call_fastpath+0x16/0x1b [1796917.981999] Code: 41 8d b6 cf 00 00 00 49 8d 7d 18 ff 90 d0 00 00 00 49 83 bc 24 98 c0 e0 81 00 0f 84 74 ff ff ff 66 0f 1f 84 00 00 00 00 00 f3 90 <49> 83 7d 18 00 75 f7 e9 5d ff ff ff 66 90 55 48 89 e5 66 66 66 [1796917.981999] Call Trace: [1796917.981999] [<ffffffff81046aae>] native_flush_tlb_others+0xe/0x10 [1796917.981999] [<ffffffff81046b7b>] flush_tlb_mm+0x5b/0xa0 [1796917.981999] [<ffffffff8113ba06>] tlb_flush_mmu+0x46/0x90 [1796917.981999] [<ffffffff8113ba64>] tlb_finish_mmu+0x14/0x40 [1796917.981999] [<ffffffff8113e3a7>] zap_page_range+0xb7/0xd0 [1796917.981999] [<ffffffff8113a85d>] madvise_vma+0xfd/0x140 [1796917.981999] [<ffffffff8107b917>] ? __set_task_blocked+0x37/0x80 [1796917.981999] [<ffffffff81095b27>] ? getnstimeofday+0x57/0xe0 [1796917.981999] [<ffffffff8113aa7e>] sys_madvise+0x1de/0x280 [1796917.981999] [<ffffffff81666b82>] system_call_fastpath+0x16/0x1b [1796917.992066] ata2: drained 65536 bytes to clear DRQ """ Analysis Bellow... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1467955/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp