------- Comment From [email protected] 2017-03-09 08:06 EDT------- This issue is resolved in latest 14.04 build.. closing this bug as fixed .13.0-112-generic dump.20170309072503090725# crash /usr/lib/debug/boot/vmlinux-3.
crash 7.0.3 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"... KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-112-generic DUMPFILE: dump.201703090725 [PARTIAL DUMP] CPUS: 80 DATE: Thu Mar 9 07:23:46 2017 UPTIME: 00:11:05 LOAD AVERAGE: 0.00, 0.04, 0.04 TASKS: 640 NODENAME: ltciofvtr-hab2 RELEASE: 3.13.0-112-generic VERSION: #159-Ubuntu SMP Fri Mar 3 15:29:14 UTC 2017 MACHINE: ppc64le (2926 Mhz) MEMORY: 256 GB PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log for details) PID: 1845 COMMAND: "bash" TASK: c000003f66100000 [THREAD_INFO: c000003f78200000] CPU: 20 STATE: TASK_RUNNING (PANIC) crash> bt PID: 1845 TASK: c000003f66100000 CPU: 20 COMMAND: "bash" #0 [c000003f782034a0] .crash_kexec at c00000000013875c #1 [c000003f78203690] .die at c000000000022618 #2 [c000003f78203730] .bad_page_fault at c00000000004a580 #3 [c000003f782037b0] handle_page_fault at c000000000009584 Data Access error [300] exception frame: R0: c0000000005962ac R1: c000003f78203aa0 R2: c000000001661000 R3: 0000000000000063 R4: c000000004b85bd0 R5: c000000004b964c8 R6: 00000000000002b0 R7: c0000000018575c0 R8: c000000000e21000 R9: 0000000000000001 R10: 0000000000000000 R11: 00000000000000c2 R12: 0000000042422844 R13: c000000007e44600 R14: 0000010039206d68 R15: 000000001016e048 R16: 000000001013ab40 R17: 00000100392069a8 R18: 000000001016ff10 R19: 000000001016e048 R20: 00000000100c2f80 R21: 0000000000000000 R22: 0000000010172008 R23: 000000001016f880 R24: 0000000000000004 R25: 0000000000000000 R26: 0000000000000001 R27: c0000000014e8370 R28: c0000000019a4eb8 R29: 0000000000000063 R30: c0000000014bda68 R31: c0000000014e8730 NIP: c00000000059534c MSR: 9000000000009033 OR3: c000000000009318 CTR: c000000000595320 LR: c0000000005962ac XER: 0000000020000000 CCR: 0000000042422822 MQ: 0000000000000000 DAR: 0000000000000000 DSISR: 0000000042000000 Syscall Result: 0000000000000000 #4 [c000003f78203aa0] .sysrq_handle_crash at c00000000059534c [Link Register ] [c000003f78203aa0] .__handle_sysrq at c0000000005962ac -- MORE -- forward: <SPACE>, <ENTER> or j backward: b or k quit: q .. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to crash in Ubuntu. https://bugs.launchpad.net/bugs/1607894 Title: Unable to analyse vmcore/dump via crash due to bad kernel debug info build Status in crash package in Ubuntu: Fix Released Status in crash source package in Trusty: Fix Committed Bug description: [ Test case ] * trigger a kernel crash dump * use crash tool to analyse it * ... on ppc64el architecture == Comment: #0 - NAVEED A. UPPINANGADY SALIH - 2016-07-01 02:41:07 == ---Problem Description--- Unable to analyse vmcore/kernel dumpfile generated by 4.4.0-29-generic kernel with ddeb kernel debuginfo available in http://ddebs.ubuntu.com/pool/main/l/linux/ http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-29 -generic-dbgsym_4.4.0-29.48_ppc64el.ddeb ---uname output--- 4.4.0-29-generic #48~14.04.1-Ubuntu SMP Wed Jun 29 19:55:03 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = model : 8247-22L machine : PowerNV 8247-22L ---Steps to Reproduce--- # crash /usr/lib/debug/boot/vmlinux-4.4.0-29-generic /var/crash/201606300840/dump.201606300840 crash 7.0.3 ... This GDB was configured as "powerpc64le-unknown-linux-gnu"... crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible" WARNING: cannot read cpu_possible_map crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present" WARNING: cannot read cpu_present_map crash: invalid kernel virtual address: 5a09344e320a1886 type: "online" WARNING: cannot read cpu_online_map WARNING: cannot read linux_banner string crash: /usr/lib/debug/boot/vmlinux-4.4.0-29-generic and /var/crash/201606300840/dump.201606300840 do not match! Userspace rpm: linux-image-4.4.0-29-generic- dbgsym_4.4.0-29.48_ppc64el.ddeb == Comment: #5 - Naresh Bannoth - 2016-07-27 04:12:09 == The Latest kernel I am having is as follows root@ltcalpine-lp6:~# uname -a Linux ltcalpine-lp6 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:03:56 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux root@ltcalpine-lp6:~# ====>>> the issue same as posted originally. ... This GDB was configured as "powerpc64le-unknown-linux-gnu"... crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible" WARNING: cannot read cpu_possible_map crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present" WARNING: cannot read cpu_present_map crash: invalid kernel virtual address: 5a09344e320a1886 type: "online" WARNING: cannot read cpu_online_map WARNING: cannot read linux_banner string crash: /usr/lib/debug/boot/vmlinux-4.4.0-31-generic and /var/crash/201607270248/dump.201607270248 do not match! == Comment: #6 - Hari Krishna Bathini - 2016-07-28 11:49:31 == The kernel is compiled with gcc version 4.8.4 while the vmlinux pulled from http://ddebs.ubuntu.com/pool/main/l/linux/ is compiled with gcc version 5.3.1 which is the source of the issue. Resolved this by pulling the kernel debug symbols package from: deb http://ddebs.ubuntu.com/ -updates main which indeed has the vmlinux compiled with gcc version 4.8.4. But I got this error: crash: invalid structure member offset: module_num_symtab FILE: kernel.c LINE: 3049 FUNCTION: module_init() after the update owning to couple of missing patches listed below: commit 6f1f78e33474d00d5f261d7ed9d835c558b34d61 Author: Dave Anderson <[email protected]> Date: Wed Jan 20 09:56:36 2016 -0500 Fix for the changes made to the kernel module structure introduced by this kernel commit for Linux 4.5 and later kernels: commit 7523e4dc5057e157212b4741abd6256e03404cf1 module: use a structure to encapsulate layout. Without the patch, the crash session fails during initialization with the error message: "crash: invalid structure member offset: module_init_text_size". ([email protected]) commit 098cdab16dfa6a85e9dad2cad604dee14ee15f66 Author: Dave Anderson <[email protected]> Date: Fri Feb 12 14:32:53 2016 -0500 Fix for the changes made to the kernel module structure introduced by this kernel commit for Linux 4.5 and later kernels: commit 8244062ef1e54502ef55f54cced659913f244c3e modules: fix longstanding /proc/kallsyms vs module insertion race. Without the patch, the crash session fails during initialization with the error message: "crash: invalid structure member offset: module_num_symtab". ([email protected]) Applying the above patches on top of crash tool version 7.0.3-3ubuntu4.4, was able to analyze the dump. The patches apply cleanly.. Thanks Hari == Comment: #8 - Naresh Bannoth - 2016-07-29 06:59:23 == Verified with 4.4.0-31-generic kernel. on Applying the patches mentioned in the #C 6, it is working fine. I took the Compiled Binary attached in the BUG and able to analyse the Dumpfile without any issues. Thanks Naresh == Comment: #9 - Hari Krishna Bathini - 2016-07-29 07:47:25 == To sum this up, two things to make this work: 1. Install debug symbols package from here deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse deb http://ddebs.ubuntu.com/ trusty-security main restricted universe multiverse deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe multiverse instead of http://ddebs.ubuntu.com/pool/main/l/linux/ as the gcc versions don't match. 2. Use a crash binary that has the upstream patches 6f1f78 & 098cda Thanks Hari To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/crash/+bug/1607894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
