** Tags added: ppc64el-kdump
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1680349
Title:
Ubuntu 17.04: Kdump fails to capture dump on Firestone NV when machine
crashes while running stress-ng.
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Bug description:
== Comment: #0 - PAVITHRA R. PRAKASH <> - 2017-03-10 02:43:10 ==
---Problem Description---
Ubuntu 17.04: Kdump fails to capture dump on Firestone NV when machine
crashes while running stress-ng. Machine hangs.
---Steps to Reproduce---
1. Configure kdump.
2. Install stress-ng
# apt-get install stress-ng
3. Run stress-ng
# stress-ng - a 0
Logs:
========
root@ltc-firep3:~# kdump-config load
Modified cmdline:root=UUID=8b0d5b99-6087-4f40-82ea-375c83a4c139 ro quiet
splash irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
ata_piix.prefer_ms_hyperv=0 elfcorehdr=155200K
* loaded kdump kernel
root@ltc-firep3:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.10.0-11-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to
/var/lib/kdump/initrd.img-4.10.0-11-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p
--command-line="root=UUID=8b0d5b99-6087-4f40-82ea-375c83a4c139 ro quiet splash
irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img
/var/lib/kdump/vmlinuz
root@ltc-firep3:~# stress-ng -a 0
stress-ng: info: [3900] defaulting to a 86400 second run per stressor
stress-ng: info: [3900] dispatching hogs: 160 af-alg, 160 affinity, 160 aio,
160 aiol, 160 apparmor, 160 atomic, 160 bigheap, 160 brk, 160 bsearch, 160
cache, 160 cap, 160 chdir, 160 chmod, 160 chown, 160 chroot, 160 clock, 160
clone, 160 context, 160 copy-file, 160 cpu, 160 cpu-online, 160 crypt, 160
daemon, 160 dccp, 160 dentry, 160 dir, 160 dirdeep, 160 dnotify, 160 dup, 160
epoll, 160 eventfd, 160 exec, 160 fallocate, 160 fanotify, 160 fault, 160
fcntl, 160 fiemap, 160 fifo, 160 filename, 160 flock, 160 fork, 160 fp-error,
160 fstat, 160 full, 160 futex, 160 get, 160 getdent, 160 getrandom, 160
handle, 160 hdd, 160 heapsort, 160 hsearch, 160 icache, 160 icmp-flood, 160
inotify, 160 io, 160 iomix, 160 ioprio, 160 itimer, 160 kcmp, 160 key, 160
kill, 160 klog, 160 lease, 160 link, 160 locka, 160 lockbus, 160 lockf, 160
lockofd, 160 longjmp, 160 lsearch, 160 madvise, 160 malloc, 160 matrix, 160
membarrier, 160 memcpy, 160 memfd, 160 mergesort, 160 mincore, 160 mknod, 160
mlock, 160 mmap, 160 mmapfork, 160 mmapmany, 160 mq, 160 mremap, 160 msg, 160
msync, 160 netlink-proc, 160 nice, 160 nop, 160 null, 160 numa, 160 oom-pipe,
160 opcode, 160 open, 160 personality, 160 pipe, 160 poll, 160 procfs, 160
pthread, 160 ptrace, 160 pty, 160 qsort, 160 quota, 160 rdrand, 160 readahead,
160 remap, 160 rename, 160 resources, 160 rlimit, 160 rmap, 160 rtc, 160
schedpolicy, 160 sctp, 160 seal, 160 seccomp, 160 seek, 160 sem, 160 sem-sysv,
160 sendfile, 160 shm, 160 shm-sysv, 160 sigfd, 160 sigfpe, 160 sigpending, 160
sigq, 160 sigsegv, 160 sigsuspend, 160 sleep, 160 sock, 160 sockfd, 160
sockpair, 160 spawn, 160 splice, 160 stack, 160 stackmmap, 160 str, 160 stream,
160 switch, 160 symlink, 160 sync-file, 160 sysfs, 160 sysinfo, 160 tee, 160
timer, 160 timerfd, 160 tlb-shootdown, 160 tmpfs, 160 tsc, 160 tsearch, 160
udp, 160 udp-flood, 160 unshare, 160 urandom, 160 userfaultfd, 160 utime, 160
vecmath, 160 vfork, 160 vforkmany, 160 vm, 160 vm-rw, 160 vm-splice, 160 wait,
160 wcs, 160 xattr, 160 yield, 160 zero, 160 zlib, 160 zombie
stress-ng: info: [3900] cache allocate: using built-in defaults as unable to
determine cache details
stress-ng: info: [3900] cache allocate: default cache size: 2048K
stress-ng: info: [3907] stress-ng-atomic: this stressor is not implemented
on this system: ppc64le Linux 4.10.0-11-generic
stress-ng: info: [3955] stress-ng-exec: running as root, won't run test.
stress-ng: info: [3999] stress-ng-icache: this stressor is not implemented
on this system: ppc64le Linux 4.10.0-11-generic
stress-ng: info: [4040] stress-ng-lockbus: this stressor is not implemented
on this system: ppc64le Linux 4.10.0-11-generic
stress-ng: info: [4313] stress-ng-numa: system has 2 of a maximum 256 memory
NUMA nodes
stress-ng: info: [4455] stress-ng-rdrand: this stressor is not implemented
on this system: ppc64le Linux 4.10.0-11-generic
stress-ng: fail: [4558] stress-ng-rtc: ioctl RTC_ALRM_READ failed, errno=22
(Invalid argument)
stress-ng: fail: [4017] stress-ng-key: keyctl KEYCTL_DESCRIBE failed,
errno=127 (Key has expired)
stress-ng: fail: [4017] stress-ng-key: keyctl KEYCTL_UPDATE failed,
errno=127 (Key has expired)
stress-ng: fail: [4017] stress-ng-key: keyctl KEYCTL_READ failed, errno=127
(Key has expired)
stress-ng: fail: [4017] stress-ng-key: request_key failed, errno=126
(Required key not available)
stress-ng: fail: [4017] stress-ng-key: keyctl KEYCTL_DESCRIBE failed,
errno=127 (Key has expired)
info: 5 failures reached, aborting stress process
[ 170.733680] Memory failure: 0xceda8: recovery action for dirty LRU page:
Recovered
[ 171.036660] Memory failure: 0xce8e9: recovery action for dirty LRU page:
Recovered
[ 171.161610] Memory failure: 0xce4fb: recovery action for dirty LRU page:
Recovered
[ 171.170348] AppArmor DFA next/check upper bounds error
[ 171.204790] Memory failure: 0xd2146: recovery action for dirty LRU page:
Recovered
[ 171.232026] Memory failure: 0xcefe6: recovery action for dirty LRU page:
Recovered
[ 171.232899] Memory failure: 0xce578: recovery action for dirty LRU page:
Recovered
[ 171.236850] Memory failure: 0xcfdfb: recovery action for dirty LRU page:
Recovered
[ 171.336249] Memory failure: 0xcd715: recovery action for dirty LRU page:
Recovered
[ 171.337550] Memory failure: 0xfb86c: recovery action for dirty LRU page:
Recovered
[ 171.367483] Memory failure: 0xce92c: recovery action for dirty LRU page:
Recovered
[ 171.369980] Memory failure: 0xceabe: recovery action for dirty LRU page:
Recovered
[ 171.372534] Memory failure: 0xbcf3a: recovery action for dirty LRU page:
Recovered
[ 171.375318] Memory failure: 0xceef9: recovery action for dirty LRU page:
Recovered
[ 171.377701] Memory failure: 0xce722: recovery action for dirty LRU page:
Recovered
[ 171.384725] Memory failure: 0xcedef: recovery action for dirty LRU page:
Recovered
[ 171.398538] Memory failure: 0xcf927: recovery action for dirty LRU page:
Recovered
[ 171.401492] Memory failure: 0xce881: recovery action for dirty LRU page:
Recovered
[ 171.403476] Memory failure: 0xce2d4: recovery action for dirty LRU page:
Recovered
[ 171.404104] Memory failure: 0xce17a: recovery action for dirty LRU page:
Recovered
[ 171.404682] Memory failure: 0xd9f0b: recovery action for dirty LRU page:
Recovered
stress-ng: info: [4865] stress-ng-spawn: running as root, won't run test.
[ 171.406159] Memory failure: 0xdaae0: recovery action for dirty LRU page:
Recovered
[ 171.415810] Memory failure: 0xb5355: recovery action for dirty LRU page:
Recovered
[ 171.434513] Memory failure: 0xb5576: recovery action for dirty LRU page:
Recovered
[ 171.435161] Memory failure: 0xbd0fd: recovery action for dirty LRU page:
Recovered
[ 171.436046] Memory failure: 0xceec0: recovery action for dirty LRU page:
Recovered
[ 171.449215] Memory failure: 0xcecda: recovery action for dirty LRU page:
Recovered
[ 171.453705] Memory failure: 0xcf005: recovery action for dirty LRU page:
Recovered
[ 171.491202] Memory failure: 0xfb99e: recovery action for dirty LRU page:
Recovered
[ 171.493054] Memory failure: 0xb2dbe: recovery action for dirty LRU page:
Recovered
[ 171.503540] Memory failure: 0xced0f: recovery action for dirty LRU page:
Recovered
[ 171.504809] Memory failure: 0xb2dad: recovery action for clean LRU page:
Recovered
[ 171.506327] Memory failure: 0xb3268: recovery action for dirty LRU page:
Recovered
[ 171.523449] Memory failure: 0xb3238: recovery action for dirty LRU page:
Recovered
[ 171.524558] Memory failure: 0xcea57: recovery action for dirty LRU page:
Recovered
[ 171.525611] Memory failure: 0xce6c8: recovery action for dirty LRU page:
Recovered
[ 171.526501] Memory failure: 0xbd0d0: recovery action for dirty LRU page:
Recovered
[ 171.528740] Memory failure: 0xcea27: recovery action for dirty LRU page:
Recovered
[ 171.536166] Memory failure: 0xce469: recovery action for dirty LRU page:
Recovered
[ 171.537409] Memory failure: 0xcec3f: recovery action for dirty LRU page:
Recovered
[ 171.538991] Memory failure: 0xcec80: recovery action for dirty LRU page:
Recovered
[ 171.540183] Memory failure: 0xb0283: recovery action for dirty LRU page:
Recovered
[ 171.568190] Memory failure: 0xb0165: recovery action for dirty LRU page:
Recovered
[ 171.569451] Memory failure: 0xda648: recovery action for dirty LRU page:
Recovered
[ 171.669472] Memory failure: 0xb2d6a: recovery action for dirty LRU page:
Recovered
stress-ng: info: [4929] stress-ng-stream: using built-in defaults as unable
to determine cache details
stress-ng: info: [4929] stress-ng-stream: stressor loosely based on a
variant of the STREAM benchmark code
stress-ng: info: [4929] stress-ng-stream: do NOT submit any of these results
to the STREAM benchmark results
stress-ng: info: [4929] stress-ng-stream: Using CPU cache size of 2048K
[ 171.722081] Memory failure: 0xcf20d: recovery action for dirty LRU page:
Recovered
[ 171.723615] Memory failure: 0xa975f: recovery action for dirty LRU page:
Recovered
[ 171.745730] Memory failure: 0xb2d85: recovery action for clean LRU page:
Recovered
stress-ng: info: [4986] stress-ng-sysfs: running as root, just traversing
/sys and not read/writing to /sys files.
[ 172.043162] Memory failure: 0xaa6aa: recovery action for dirty LRU page:
Recovered
[ 172.048888] Memory failure: 0xb02d0: recovery action for dirty LRU page:
Recovered
[ 172.103892] Memory failure: 0xcd8db: recovery action for dirty LRU page:
Recovered
[ 172.105545] Memory failure: 0xb2d9e: recovery action for dirty LRU page:
Recovered
[ 172.106053] Memory failure: 0xcf2f4: recovery action for dirty LRU page:
Recovered
[ 172.106224] Memory failure: 0xa9758: recovery action for clean LRU page:
Recovered
[ 172.146851] Memory failure: 0xce5e8: recovery action for clean LRU page:
Recovered
[ 172.234564] Memory failure: 0x9e8b2: recovery action for dirty LRU page:
Recovered
[ 172.236835] Memory failure: 0xac4f8: recovery action for dirty LRU page:
Recovered
[ 172.238363] Memory failure: 0xcebb2: recovery action for dirty LRU page:
Recovered
stress-ng: info: [5105] stress-ng-tsc: this stressor is not implemented on
this system: ppc64le Linux 4.10.0-11-generic
[ 172.494650] Memory failure: 0xcecb6: recovery action for clean LRU page:
Recovered
[ 172.495944] Memory failure: 0xa9710: recovery action for dirty LRU page:
Recovered
[ 172.496511] Memory failure: 0xb55d7: recovery action for dirty LRU page:
Recovered
[ 172.496932] Memory failure: 0x9e8cb: recovery action for dirty LRU page:
Recovered
[ 172.716658] Memory failure: 0x9e628: recovery action for dirty LRU page:
Recovered
[ 172.780960] Memory failure: 0xcf3ac: recovery action for dirty LRU page:
Recovered
[ 172.781447] Memory failure: 0xceaac: recovery action for dirty LRU page:
Recovered
[ 172.781891] Memory failure: 0xb55a1: recovery action for dirty LRU page:
Recovered
[ 172.845268] Memory failure: 0x84318: recovery action for dirty LRU page:
Recovered
[ 172.846308] Memory failure: 0x84322: recovery action for dirty LRU page:
Recovered
[ 172.860021] Memory failure: 0xbd067: recovery action for dirty LRU page:
Recovered
[ 172.924176] Memory failure: 0xce68e: recovery action for dirty LRU page:
Recovered
[ 172.926255] Memory failure: 0x92ee8: recovery action for dirty LRU page:
Recovered
[ 172.926720] Memory failure: 0xda136: recovery action for dirty LRU page:
Recovered
[ 172.927534] Memory failure: 0xb2d75: recovery action for dirty LRU page:
Recovered
[ 173.008909] Memory failure: 0xac4e6: recovery action for dirty LRU page:
Recovered
[ 173.042161] Memory failure: 0xcea49: recovery action for dirty LRU page:
Recovered
[ 173.076591] Memory failure: 0x9e8fb: recovery action for dirty LRU page:
Recovered
[ 173.124359] Memory failure: 0x8434b: recovery action for dirty LRU page:
Recovered
[ 173.288102] Memory failure: 0xcf5e7: recovery action for dirty LRU page:
Recovered
[ 173.440243] Memory failure: 0xb012d: recovery action for dirty LRU page:
Recovered
[ 173.565679] Memory failure: 0x1cc382: recovery action for clean LRU page:
Recovered
[ 173.620166] Memory failure: 0x84334: recovery action for dirty LRU page:
Recovered
[ 173.635189] Memory failure: 0xb02bf: recovery action for dirty LRU page:
Recovered
[ 173.636070] Memory failure: 0x9e8f0: recovery action for dirty LRU page:
Recovered
[ 173.638929] Memory failure: 0x84362: recovery action for dirty LRU page:
Recovered
[ 173.643249] Memory failure: 0xcda1c: recovery action for dirty LRU page:
Recovered
[ 173.648607] Memory failure: 0x9a27a: recovery action for dirty LRU page:
Recovered
[ 173.651927] Memory failure: 0xced46: recovery action for dirty LRU page:
Recovered
[ 173.711413] Memory failure: 0x9a270: recovery action for dirty LRU page:
Recovered
[ 173.733759] Memory failure: 0xb55b1: recovery action for dirty LRU page:
Recovered
[ 173.738553] Memory failure: 0x840d1: recovery action for dirty LRU page:
Recovered
[ 173.740023] Memory failure: 0xb01ae: recovery action for dirty LRU page:
Recovered
[ 173.740992] Memory failure: 0x1c9ca8: recovery action for dirty LRU page:
Recovered
[ 173.742282] Memory failure: 0xa97fa: recovery action for dirty LRU page:
Recovered
[ 173.783778] Memory failure: 0xc763c: recovery action for dirty LRU page:
Recovered
[ 173.785593] Memory failure: 0xb02b6: recovery action for dirty LRU page:
Recovered
[ 173.788206] AppArmor DFA next/check upper bounds error
[ 173.788390] Memory failure: 0x1ca066: dirty LRU page still referenced by 1
users
[ 173.788395] Memory failure: 0x1ca066: recovery action for dirty LRU page:
Failed
[ 174.403722] Memory failure: 0x1c979a: recovery action for dirty LRU page:
Recovered
[ 174.428211] Memory failure: 0xb02db: recovery action for dirty LRU page:
Recovered
stress-ng: info: [5591] stress-ng-yield: limiting to 160 yielders (instance
0)
stress-ng: info: [5689] stress-ng-atomic: this stressor is not implemented
on this system: ppc64le Linux 4.10.0-11-generic
[ 174.643022] Memory failure: 0x1ca7c1: recovery action for dirty LRU page:
Recovered
stress-ng: info: [6033] stress-ng-exec: running as root, won't run test.
[ 174.691794] Unable to handle kernel paging request for data at address
0x000002f4
[ 174.692217] Faulting instruction address: 0xd000000014bc0a90
[ 174.692484] Oops: Kernel access of bad area, sig: 11 [#1]
[ 174.692780] SMP NR_CPUS=2048
[ 174.692788] NUMA
[ 174.693003] PowerNV
[ 174.693269] Modules linked in: btrfs xor raid6_pq cuse wp512 kvm_hv kvm_pr
sctp(+) rmd320 libcrc32c dccp_ipv4(+) kvm rmd256 rmd160 rmd128 md4 binfmt_misc
algif_hash dccp af_alg ofpart cmdlinepart ipmi_powernv ipmi_devintf
powernv_flash ipmi_msghandler mtd opal_prd ibmpowernv powernv_rng joydev
input_leds mac_hid at24 nvmem_core uio_pdrv_genirq uio ip_tables x_tables
autofs4 hid_generic usbhid hid uas usb_storage ast crc32c_vpmsum i2c_algo_bit
ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ahci
libahci tg3
[ 174.694545] CPU: 98 PID: 6170 Comm: stress-ng-dccp Not tainted
4.10.0-11-generic #13-Ubuntu
[ 174.694645] task: c000001e42c50800 task.stack: c000001e42cf8000
[ 174.694758] NIP: d000000014bc0a90 LR: d000000014bc21cc CTR:
c000000000a3b340
[ 174.694872] REGS: c000001fff4476b0 TRAP: 0300 Not tainted
(4.10.0-11-generic)
[ 174.694974] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
[ 174.695059] CR: 24002242 XER: 20000000
[ 174.695217] CFAR: c000000000008850 DAR: 00000000000002f4 DSISR: 40000000
SOFTE: 1
[ 174.695217] GPR00: d000000014bc21cc c000001fff447930 d000000014bcb670
0000000000000001
[ 174.695217] GPR04: c000001e3b602700 c000001df14a0c60 0000000000000474
c000001df14a0c74
[ 174.695217] GPR08: c000001df14a0800 0000000000000000 c000001e3b60c400
0000000000000000
[ 174.695217] GPR12: 0000000000002200 c000000007b87200 c000001fff444000
0000000000000000
[ 174.695217] GPR16: c000000000fc2800 0000000000000000 0000000000000040
0000000000002711
[ 174.695217] GPR20: 000000000000a4d2 000000000100007f 000000000100007f
c0000000013d2880
[ 174.695217] GPR24: 0000000000000001 0000000000000001 0000000000000000
0000000000000004
[ 174.695217] GPR28: c0000000013d2880 c000001df14a0c74 0000000000000000
c000001e3b602700
[ 174.696147] NIP [d000000014bc0a90] dccp_v4_ctl_send_reset+0xa8/0x2f0
[dccp_ipv4]
[ 174.696238] LR [d000000014bc21cc] dccp_v4_rcv+0x5d4/0x850 [dccp_ipv4]
[ 174.696312] Call Trace:
[ 174.696345] [c000001fff447930] [c000001fff4479c0] 0xc000001fff4479c0
(unreliable)
[ 174.696978] [c000001fff4479c0] [d000000
-----------------------------MACHINE HANGS
-------------------------------------
== Comment: #29 - Kevin W. Rudd <> - 2017-03-20 12:50:22 ==
Hari.
I was able to get access to the system for a quick set of validation
tests. With the default kdump settings, kdump completed OK and
correctly saved a vmcore. When stress-ng is run first, kdump hangs
with the default settings, and also when the settings are modified to
use "maxcpus=1" and "noirqdistrib".
The following message was printed prior to the stress-ng induced
hangs, but not when kdump completed without stress-ng running:
"Ignoring boot flags, incorrect version 0x0"
I will attach console logs from each test for review.
== Comment: #30 - Kevin W. Rudd <> - 2017-03-20 12:51:54 ==
The default boot options had "quiet splash", so I trimmed out the useless
"ubuntu" splash messages from the log.
== Comment: #36 - Hari Krishna Bathini <> - 2017-04-05 13:31:23 ==
If panic timeout is set to zero and any secondary CPUs don't respond to IPI,
kdump waits forever for a system reset, to try again to get ALL secondary
CPUs to respond to IPI. The hang here is because panic timeout is set to
zero and a few secondary CPUs didn't respond to IPI. System reset support
is still work in progress for Open Power machines. Meantime, to workaround
the hang issue, panic timeout value can be set to a non-zero value with
$ echo 10 > /proc/sys/kernel/panic
I did try it but kdump didn't take off. Instead the system just rebooted
(better than a hang, I guess :) ). As for why kdump didn't take off, I
am debugging it..
Hi Canonical,
I think it would be better to have a non-zero default value for panic
timeout (CONFIG_PANIC_TIMEOUT)?
Thanks
Hari
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1680349/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
