Default Comment by Bridge

** Attachment added: "Failed kdump with defaults after stress-ng was invoked"
   
https://bugs.launchpad.net/bugs/1680349/+attachment/5077999/+files/stress-ng_kdump_with_defaults.log

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1680349

Title:
  Kdump fails to capture dump on Firestone NV when machine crashes while
  running stress-ng.

Status in The Ubuntu-power-systems project:
  Incomplete
Status in linux package in Ubuntu:
  New

Bug description:
  == Comment: #0 - PAVITHRA R. PRAKASH <> - 2017-03-10 02:43:10 ==
  ---Problem Description---

  Ubuntu 17.04: Kdump fails to capture dump on Firestone NV when machine
  crashes while running stress-ng. Machine hangs.

  ---Steps to Reproduce---

  1. Configure kdump.
  2. Install stress-ng
  # apt-get install stress-ng
  3. Run stress-ng
  # stress-ng - a 0

  
  Logs:
  ========
  root@ltc-firep3:~# kdump-config load
  Modified cmdline:root=UUID=8b0d5b99-6087-4f40-82ea-375c83a4c139 ro quiet 
splash irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service 
ata_piix.prefer_ms_hyperv=0 elfcorehdr=155200K 
   * loaded kdump kernel
  root@ltc-firep3:~# kdump-config show
  DUMP_MODE:        kdump
  USE_KDUMP:        1
  KDUMP_SYSCTL:     kernel.panic_on_oops=1
  KDUMP_COREDIR:    /var/crash
  crashkernel addr: 
     /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.10.0-11-generic
  kdump initrd: 
     /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-4.10.0-11-generic
  current state:    ready to kdump

  kexec command:
    /sbin/kexec -p 
--command-line="root=UUID=8b0d5b99-6087-4f40-82ea-375c83a4c139 ro quiet splash 
irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service 
ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img 
/var/lib/kdump/vmlinuz
  root@ltc-firep3:~# stress-ng -a 0
  stress-ng: info:  [3900] defaulting to a 86400 second run per stressor
  stress-ng: info:  [3900] dispatching hogs: 160 af-alg, 160 affinity, 160 aio, 
160 aiol, 160 apparmor, 160 atomic, 160 bigheap, 160 brk, 160 bsearch, 160 
cache, 160 cap, 160 chdir, 160 chmod, 160 chown, 160 chroot, 160 clock, 160 
clone, 160 context, 160 copy-file, 160 cpu, 160 cpu-online, 160 crypt, 160 
daemon, 160 dccp, 160 dentry, 160 dir, 160 dirdeep, 160 dnotify, 160 dup, 160 
epoll, 160 eventfd, 160 exec, 160 fallocate, 160 fanotify, 160 fault, 160 
fcntl, 160 fiemap, 160 fifo, 160 filename, 160 flock, 160 fork, 160 fp-error, 
160 fstat, 160 full, 160 futex, 160 get, 160 getdent, 160 getrandom, 160 
handle, 160 hdd, 160 heapsort, 160 hsearch, 160 icache, 160 icmp-flood, 160 
inotify, 160 io, 160 iomix, 160 ioprio, 160 itimer, 160 kcmp, 160 key, 160 
kill, 160 klog, 160 lease, 160 link, 160 locka, 160 lockbus, 160 lockf, 160 
lockofd, 160 longjmp, 160 lsearch, 160 madvise, 160 malloc, 160 matrix, 160 
membarrier, 160 memcpy, 160 memfd, 160 mergesort, 160 mincore, 160 mknod, 160 
mlock, 160 mmap, 160 mmapfork, 160 mmapmany, 160 mq, 160 mremap, 160 msg, 160 
msync, 160 netlink-proc, 160 nice, 160 nop, 160 null, 160 numa, 160 oom-pipe, 
160 opcode, 160 open, 160 personality, 160 pipe, 160 poll, 160 procfs, 160 
pthread, 160 ptrace, 160 pty, 160 qsort, 160 quota, 160 rdrand, 160 readahead, 
160 remap, 160 rename, 160 resources, 160 rlimit, 160 rmap, 160 rtc, 160 
schedpolicy, 160 sctp, 160 seal, 160 seccomp, 160 seek, 160 sem, 160 sem-sysv, 
160 sendfile, 160 shm, 160 shm-sysv, 160 sigfd, 160 sigfpe, 160 sigpending, 160 
sigq, 160 sigsegv, 160 sigsuspend, 160 sleep, 160 sock, 160 sockfd, 160 
sockpair, 160 spawn, 160 splice, 160 stack, 160 stackmmap, 160 str, 160 stream, 
160 switch, 160 symlink, 160 sync-file, 160 sysfs, 160 sysinfo, 160 tee, 160 
timer, 160 timerfd, 160 tlb-shootdown, 160 tmpfs, 160 tsc, 160 tsearch, 160 
udp, 160 udp-flood, 160 unshare, 160 urandom, 160 userfaultfd, 160 utime, 160 
vecmath, 160 vfork, 160 vforkmany, 160 vm, 160 vm-rw, 160 vm-splice, 160 wait, 
160 wcs, 160 xattr, 160 yield, 160 zero, 160 zlib, 160 zombie
  stress-ng: info:  [3900] cache allocate: using built-in defaults as unable to 
determine cache details
  stress-ng: info:  [3900] cache allocate: default cache size: 2048K
  stress-ng: info:  [3907] stress-ng-atomic: this stressor is not implemented 
on this system: ppc64le Linux 4.10.0-11-generic
  stress-ng: info:  [3955] stress-ng-exec: running as root, won't run test.
  stress-ng: info:  [3999] stress-ng-icache: this stressor is not implemented 
on this system: ppc64le Linux 4.10.0-11-generic
  stress-ng: info:  [4040] stress-ng-lockbus: this stressor is not implemented 
on this system: ppc64le Linux 4.10.0-11-generic
  stress-ng: info:  [4313] stress-ng-numa: system has 2 of a maximum 256 memory 
NUMA nodes
  stress-ng: info:  [4455] stress-ng-rdrand: this stressor is not implemented 
on this system: ppc64le Linux 4.10.0-11-generic
  stress-ng: fail:  [4558] stress-ng-rtc: ioctl RTC_ALRM_READ failed, errno=22 
(Invalid argument)
  stress-ng: fail:  [4017] stress-ng-key: keyctl KEYCTL_DESCRIBE failed, 
errno=127 (Key has expired)
  stress-ng: fail:  [4017] stress-ng-key: keyctl KEYCTL_UPDATE failed, 
errno=127 (Key has expired)
  stress-ng: fail:  [4017] stress-ng-key: keyctl KEYCTL_READ failed, errno=127 
(Key has expired)
  stress-ng: fail:  [4017] stress-ng-key: request_key failed, errno=126 
(Required key not available)
  stress-ng: fail:  [4017] stress-ng-key: keyctl KEYCTL_DESCRIBE failed, 
errno=127 (Key has expired)
  info: 5 failures reached, aborting stress process
  [  170.733680] Memory failure: 0xceda8: recovery action for dirty LRU page: 
Recovered
  [  171.036660] Memory failure: 0xce8e9: recovery action for dirty LRU page: 
Recovered
  [  171.161610] Memory failure: 0xce4fb: recovery action for dirty LRU page: 
Recovered
  [  171.170348] AppArmor DFA next/check upper bounds error
  [  171.204790] Memory failure: 0xd2146: recovery action for dirty LRU page: 
Recovered
  [  171.232026] Memory failure: 0xcefe6: recovery action for dirty LRU page: 
Recovered
  [  171.232899] Memory failure: 0xce578: recovery action for dirty LRU page: 
Recovered
  [  171.236850] Memory failure: 0xcfdfb: recovery action for dirty LRU page: 
Recovered
  [  171.336249] Memory failure: 0xcd715: recovery action for dirty LRU page: 
Recovered
  [  171.337550] Memory failure: 0xfb86c: recovery action for dirty LRU page: 
Recovered
  [  171.367483] Memory failure: 0xce92c: recovery action for dirty LRU page: 
Recovered
  [  171.369980] Memory failure: 0xceabe: recovery action for dirty LRU page: 
Recovered
  [  171.372534] Memory failure: 0xbcf3a: recovery action for dirty LRU page: 
Recovered
  [  171.375318] Memory failure: 0xceef9: recovery action for dirty LRU page: 
Recovered
  [  171.377701] Memory failure: 0xce722: recovery action for dirty LRU page: 
Recovered
  [  171.384725] Memory failure: 0xcedef: recovery action for dirty LRU page: 
Recovered
  [  171.398538] Memory failure: 0xcf927: recovery action for dirty LRU page: 
Recovered
  [  171.401492] Memory failure: 0xce881: recovery action for dirty LRU page: 
Recovered
  [  171.403476] Memory failure: 0xce2d4: recovery action for dirty LRU page: 
Recovered
  [  171.404104] Memory failure: 0xce17a: recovery action for dirty LRU page: 
Recovered
  [  171.404682] Memory failure: 0xd9f0b: recovery action for dirty LRU page: 
Recovered
  stress-ng: info:  [4865] stress-ng-spawn: running as root, won't run test.
  [  171.406159] Memory failure: 0xdaae0: recovery action for dirty LRU page: 
Recovered
  [  171.415810] Memory failure: 0xb5355: recovery action for dirty LRU page: 
Recovered
  [  171.434513] Memory failure: 0xb5576: recovery action for dirty LRU page: 
Recovered
  [  171.435161] Memory failure: 0xbd0fd: recovery action for dirty LRU page: 
Recovered
  [  171.436046] Memory failure: 0xceec0: recovery action for dirty LRU page: 
Recovered
  [  171.449215] Memory failure: 0xcecda: recovery action for dirty LRU page: 
Recovered
  [  171.453705] Memory failure: 0xcf005: recovery action for dirty LRU page: 
Recovered
  [  171.491202] Memory failure: 0xfb99e: recovery action for dirty LRU page: 
Recovered
  [  171.493054] Memory failure: 0xb2dbe: recovery action for dirty LRU page: 
Recovered
  [  171.503540] Memory failure: 0xced0f: recovery action for dirty LRU page: 
Recovered
  [  171.504809] Memory failure: 0xb2dad: recovery action for clean LRU page: 
Recovered
  [  171.506327] Memory failure: 0xb3268: recovery action for dirty LRU page: 
Recovered
  [  171.523449] Memory failure: 0xb3238: recovery action for dirty LRU page: 
Recovered
  [  171.524558] Memory failure: 0xcea57: recovery action for dirty LRU page: 
Recovered
  [  171.525611] Memory failure: 0xce6c8: recovery action for dirty LRU page: 
Recovered
  [  171.526501] Memory failure: 0xbd0d0: recovery action for dirty LRU page: 
Recovered
  [  171.528740] Memory failure: 0xcea27: recovery action for dirty LRU page: 
Recovered
  [  171.536166] Memory failure: 0xce469: recovery action for dirty LRU page: 
Recovered
  [  171.537409] Memory failure: 0xcec3f: recovery action for dirty LRU page: 
Recovered
  [  171.538991] Memory failure: 0xcec80: recovery action for dirty LRU page: 
Recovered
  [  171.540183] Memory failure: 0xb0283: recovery action for dirty LRU page: 
Recovered
  [  171.568190] Memory failure: 0xb0165: recovery action for dirty LRU page: 
Recovered
  [  171.569451] Memory failure: 0xda648: recovery action for dirty LRU page: 
Recovered
  [  171.669472] Memory failure: 0xb2d6a: recovery action for dirty LRU page: 
Recovered
  stress-ng: info:  [4929] stress-ng-stream: using built-in defaults as unable 
to determine cache details
  stress-ng: info:  [4929] stress-ng-stream: stressor loosely based on a 
variant of the STREAM benchmark code
  stress-ng: info:  [4929] stress-ng-stream: do NOT submit any of these results 
to the STREAM benchmark results
  stress-ng: info:  [4929] stress-ng-stream: Using CPU cache size of 2048K
  [  171.722081] Memory failure: 0xcf20d: recovery action for dirty LRU page: 
Recovered
  [  171.723615] Memory failure: 0xa975f: recovery action for dirty LRU page: 
Recovered
  [  171.745730] Memory failure: 0xb2d85: recovery action for clean LRU page: 
Recovered
  stress-ng: info:  [4986] stress-ng-sysfs: running as root, just traversing 
/sys and not read/writing to /sys files.
  [  172.043162] Memory failure: 0xaa6aa: recovery action for dirty LRU page: 
Recovered
  [  172.048888] Memory failure: 0xb02d0: recovery action for dirty LRU page: 
Recovered
  [  172.103892] Memory failure: 0xcd8db: recovery action for dirty LRU page: 
Recovered
  [  172.105545] Memory failure: 0xb2d9e: recovery action for dirty LRU page: 
Recovered
  [  172.106053] Memory failure: 0xcf2f4: recovery action for dirty LRU page: 
Recovered
  [  172.106224] Memory failure: 0xa9758: recovery action for clean LRU page: 
Recovered
  [  172.146851] Memory failure: 0xce5e8: recovery action for clean LRU page: 
Recovered
  [  172.234564] Memory failure: 0x9e8b2: recovery action for dirty LRU page: 
Recovered
  [  172.236835] Memory failure: 0xac4f8: recovery action for dirty LRU page: 
Recovered
  [  172.238363] Memory failure: 0xcebb2: recovery action for dirty LRU page: 
Recovered
  stress-ng: info:  [5105] stress-ng-tsc: this stressor is not implemented on 
this system: ppc64le Linux 4.10.0-11-generic
  [  172.494650] Memory failure: 0xcecb6: recovery action for clean LRU page: 
Recovered
  [  172.495944] Memory failure: 0xa9710: recovery action for dirty LRU page: 
Recovered
  [  172.496511] Memory failure: 0xb55d7: recovery action for dirty LRU page: 
Recovered
  [  172.496932] Memory failure: 0x9e8cb: recovery action for dirty LRU page: 
Recovered
  [  172.716658] Memory failure: 0x9e628: recovery action for dirty LRU page: 
Recovered
  [  172.780960] Memory failure: 0xcf3ac: recovery action for dirty LRU page: 
Recovered
  [  172.781447] Memory failure: 0xceaac: recovery action for dirty LRU page: 
Recovered
  [  172.781891] Memory failure: 0xb55a1: recovery action for dirty LRU page: 
Recovered
  [  172.845268] Memory failure: 0x84318: recovery action for dirty LRU page: 
Recovered
  [  172.846308] Memory failure: 0x84322: recovery action for dirty LRU page: 
Recovered
  [  172.860021] Memory failure: 0xbd067: recovery action for dirty LRU page: 
Recovered
  [  172.924176] Memory failure: 0xce68e: recovery action for dirty LRU page: 
Recovered
  [  172.926255] Memory failure: 0x92ee8: recovery action for dirty LRU page: 
Recovered
  [  172.926720] Memory failure: 0xda136: recovery action for dirty LRU page: 
Recovered
  [  172.927534] Memory failure: 0xb2d75: recovery action for dirty LRU page: 
Recovered
  [  173.008909] Memory failure: 0xac4e6: recovery action for dirty LRU page: 
Recovered
  [  173.042161] Memory failure: 0xcea49: recovery action for dirty LRU page: 
Recovered
  [  173.076591] Memory failure: 0x9e8fb: recovery action for dirty LRU page: 
Recovered
  [  173.124359] Memory failure: 0x8434b: recovery action for dirty LRU page: 
Recovered
  [  173.288102] Memory failure: 0xcf5e7: recovery action for dirty LRU page: 
Recovered
  [  173.440243] Memory failure: 0xb012d: recovery action for dirty LRU page: 
Recovered
  [  173.565679] Memory failure: 0x1cc382: recovery action for clean LRU page: 
Recovered
  [  173.620166] Memory failure: 0x84334: recovery action for dirty LRU page: 
Recovered
  [  173.635189] Memory failure: 0xb02bf: recovery action for dirty LRU page: 
Recovered
  [  173.636070] Memory failure: 0x9e8f0: recovery action for dirty LRU page: 
Recovered
  [  173.638929] Memory failure: 0x84362: recovery action for dirty LRU page: 
Recovered
  [  173.643249] Memory failure: 0xcda1c: recovery action for dirty LRU page: 
Recovered
  [  173.648607] Memory failure: 0x9a27a: recovery action for dirty LRU page: 
Recovered
  [  173.651927] Memory failure: 0xced46: recovery action for dirty LRU page: 
Recovered
  [  173.711413] Memory failure: 0x9a270: recovery action for dirty LRU page: 
Recovered
  [  173.733759] Memory failure: 0xb55b1: recovery action for dirty LRU page: 
Recovered
  [  173.738553] Memory failure: 0x840d1: recovery action for dirty LRU page: 
Recovered
  [  173.740023] Memory failure: 0xb01ae: recovery action for dirty LRU page: 
Recovered
  [  173.740992] Memory failure: 0x1c9ca8: recovery action for dirty LRU page: 
Recovered
  [  173.742282] Memory failure: 0xa97fa: recovery action for dirty LRU page: 
Recovered
  [  173.783778] Memory failure: 0xc763c: recovery action for dirty LRU page: 
Recovered
  [  173.785593] Memory failure: 0xb02b6: recovery action for dirty LRU page: 
Recovered
  [  173.788206] AppArmor DFA next/check upper bounds error
  [  173.788390] Memory failure: 0x1ca066: dirty LRU page still referenced by 1 
users
  [  173.788395] Memory failure: 0x1ca066: recovery action for dirty LRU page: 
Failed
  [  174.403722] Memory failure: 0x1c979a: recovery action for dirty LRU page: 
Recovered
  [  174.428211] Memory failure: 0xb02db: recovery action for dirty LRU page: 
Recovered
  stress-ng: info:  [5591] stress-ng-yield: limiting to 160 yielders (instance 
0)
  stress-ng: info:  [5689] stress-ng-atomic: this stressor is not implemented 
on this system: ppc64le Linux 4.10.0-11-generic
  [  174.643022] Memory failure: 0x1ca7c1: recovery action for dirty LRU page: 
Recovered
  stress-ng: info:  [6033] stress-ng-exec: running as root, won't run test.
  [  174.691794] Unable to handle kernel paging request for data at address 
0x000002f4
  [  174.692217] Faulting instruction address: 0xd000000014bc0a90
  [  174.692484] Oops: Kernel access of bad area, sig: 11 [#1]
  [  174.692780] SMP NR_CPUS=2048 
  [  174.692788] NUMA 
  [  174.693003] PowerNV
  [  174.693269] Modules linked in: btrfs xor raid6_pq cuse wp512 kvm_hv kvm_pr 
sctp(+) rmd320 libcrc32c dccp_ipv4(+) kvm rmd256 rmd160 rmd128 md4 binfmt_misc 
algif_hash dccp af_alg ofpart cmdlinepart ipmi_powernv ipmi_devintf 
powernv_flash ipmi_msghandler mtd opal_prd ibmpowernv powernv_rng joydev 
input_leds mac_hid at24 nvmem_core uio_pdrv_genirq uio ip_tables x_tables 
autofs4 hid_generic usbhid hid uas usb_storage ast crc32c_vpmsum i2c_algo_bit 
ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ahci 
libahci tg3
  [  174.694545] CPU: 98 PID: 6170 Comm: stress-ng-dccp Not tainted 
4.10.0-11-generic #13-Ubuntu
  [  174.694645] task: c000001e42c50800 task.stack: c000001e42cf8000
  [  174.694758] NIP: d000000014bc0a90 LR: d000000014bc21cc CTR: 
c000000000a3b340
  [  174.694872] REGS: c000001fff4476b0 TRAP: 0300   Not tainted  
(4.10.0-11-generic)
  [  174.694974] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
  [  174.695059]   CR: 24002242  XER: 20000000
  [  174.695217] CFAR: c000000000008850 DAR: 00000000000002f4 DSISR: 40000000 
SOFTE: 1 
  [  174.695217] GPR00: d000000014bc21cc c000001fff447930 d000000014bcb670 
0000000000000001 
  [  174.695217] GPR04: c000001e3b602700 c000001df14a0c60 0000000000000474 
c000001df14a0c74 
  [  174.695217] GPR08: c000001df14a0800 0000000000000000 c000001e3b60c400 
0000000000000000 
  [  174.695217] GPR12: 0000000000002200 c000000007b87200 c000001fff444000 
0000000000000000 
  [  174.695217] GPR16: c000000000fc2800 0000000000000000 0000000000000040 
0000000000002711 
  [  174.695217] GPR20: 000000000000a4d2 000000000100007f 000000000100007f 
c0000000013d2880 
  [  174.695217] GPR24: 0000000000000001 0000000000000001 0000000000000000 
0000000000000004 
  [  174.695217] GPR28: c0000000013d2880 c000001df14a0c74 0000000000000000 
c000001e3b602700 
  [  174.696147] NIP [d000000014bc0a90] dccp_v4_ctl_send_reset+0xa8/0x2f0 
[dccp_ipv4]
  [  174.696238] LR [d000000014bc21cc] dccp_v4_rcv+0x5d4/0x850 [dccp_ipv4]
  [  174.696312] Call Trace:
  [  174.696345] [c000001fff447930] [c000001fff4479c0] 0xc000001fff4479c0 
(unreliable)
  [  174.696978] [c000001fff4479c0] [d000000

  -----------------------------MACHINE HANGS
  -------------------------------------

  == Comment: #29 - Kevin W. Rudd <> - 2017-03-20 12:50:22 ==
  Hari.

  I was able to get access to the system for a quick set of validation
  tests.  With the default kdump settings, kdump completed OK and
  correctly saved a vmcore.  When stress-ng is run first, kdump hangs
  with the default settings, and also when the settings are modified to
  use "maxcpus=1" and "noirqdistrib".

  The following message was printed prior to the stress-ng induced
  hangs, but not when kdump completed without stress-ng running:

  "Ignoring boot flags, incorrect version 0x0"

  I will attach console logs from each test for review.

  == Comment: #30 - Kevin W. Rudd <> - 2017-03-20 12:51:54 ==
  The default boot options had "quiet splash", so I trimmed out the useless 
"ubuntu" splash messages from the log.

  == Comment: #36 - Hari Krishna Bathini <> - 2017-04-05 13:31:23 ==
  If panic timeout is set to zero and any secondary CPUs don't respond to IPI,
  kdump waits forever for a system reset, to try again to get ALL secondary
  CPUs to respond to IPI. The hang here is because panic timeout is set to
  zero and a few secondary CPUs didn't respond to IPI. System reset support
  is still work in progress for Open Power machines. Meantime, to workaround
  the hang issue, panic timeout value can be set to a non-zero value with

          $ echo 10 > /proc/sys/kernel/panic

  I did try it but kdump didn't take off. Instead the system just rebooted
  (better than a hang, I guess :) ). As for why kdump didn't take off, I
  am debugging it..

  Hi Canonical,

  I think it would be better to have a non-zero default value for panic
  timeout (CONFIG_PANIC_TIMEOUT)?

  Thanks
  Hari

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1680349/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to