[Kernel-packages] [Bug 1658733] loglevel 8 kdump console log
Default Comment by Bridge ** Attachment added: "loglevel 8 kdump console log" https://bugs.launchpad.net/bugs/1658733/+attachment/5024207/+files/dotg9-kdump-loglevel8.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system when noirqdistrib is missing as dump kernel parameter Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Invalid Status in makedumpfile package in Ubuntu: Fix Released Status in kexec-tools source package in Trusty: New Status in makedumpfile source package in Trusty: New Status in kexec-tools source package in Xenial: New Status in makedumpfile source package in Xenial: Fix Committed Status in kexec-tools source package in Zesty: New Status in makedumpfile source package in Zesty: New Status in kexec-tools source package in Artful: Invalid Status in makedumpfile source package in Artful: Fix Committed Status in kexec-tools source package in Bionic: Invalid Status in makedumpfile source package in Bionic: Fix Released Bug description: [Impact] On Power Systems, some interrupts are missed, and dumping the crash will fail. Adding the noirqdistrib kernel parameter to the kdump kernel will fix this. [Test Case] Setting up kdump to target a virtio-scsi device on a Power System. [Regression Potential] The parameter could be interpreted differently on a different platform and kdump would fail. However, it has been verified that no other platform uses such parameter. If another parameter would have been incorrectly removed on the patch, kdump could fail on other systems. == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform
[Kernel-packages] [Bug 1658733] loglevel 8 kdump console log
Default Comment by Bridge ** Attachment added: "loglevel 8 kdump console log" https://bugs.launchpad.net/bugs/1658733/+attachment/4887387/+files/dotg9-kdump-loglevel8.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/1658733 Title: Ubuntu 16.04.2KVM:kdump fails to mount root file system on multipath root device Status in The Ubuntu-power-systems project: Confirmed Status in kexec-tools package in Ubuntu: Confirmed Status in makedumpfile package in Ubuntu: Confirmed Bug description: == Comment: #0 - Richard M. Scheller - 2016-12-14 16:50:26 == ---Problem Description--- On a KVM guest installed to a multipath root device, the kdump kernel fails to mount the root file system. This error does not occur in a similar guest installed to a single path device. Full console output of the kdump failure is attached. These messages from the output may be relevant: Begin: Loading multipath modules ... Success: loaded module dm-multipath. done. Begin: Loading multipath hardware handlers ... Failure: failed to load module sc si_dh_alua. Failure: failed to load module scsi_dh_rdac. Failure: failed to load module scsi_dh_emc. done. Begin: Starting multipathd ... done. ---uname output--- Linux dotg9 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L Ubuntu 16.04.1 KVM guest ---Steps to Reproduce--- - Install Ubuntu 16.04.1 to a muiltpath target disk - Install kdump-tools package - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg - Run update-grub - Reboot - Initiate a system crash using "echo c > /proc/sysrq-trigger" == Comment: #12 - Richard M. Scheller - 2016-12-20 20:37:45 == Here is the log level 8 kdump console log requested in comment 10. == Comment: #21 - Richard M. Scheller - 2017-01-06 11:04:17 == (In reply to comment #19) > Hi, I logged in dotkvm and I couldn't find the guest dotg9. Also, although I > found a dotg9.xml in /kte/xml/ it doesn't look like it uses multipath (it > uses .img files which I didn't found as disks). > > Could you please recreate the guest for further debug? Yes, I recreated the guest with its correct multipath lun configuration. I have also attached the guest XML to this bug. > Besides that could you please let us know: > - is the multipath the system's root? I mean / is installed/mounted on the > multipath device? Yes, the guest has only one disk. That disk is actually a LUN from a fiber channel storage device with two paths on the host side. I have passed through both paths to the guest, so the multipath nature of the target disk is known to the guest. In other words, the guest sees a multipath device and is using it as a multipath device. The root file system is called /dev/mapper/mpatha- part2 on the guest. > - how did you attach the device to the guest? Each FC LUN path on the host is mapped to a virtio-scsi controller on the guest using LUN passthrough. (See the guest XML for details on this.) == Comment: #22 - Mauro Sergio Martins Rodrigues - 2017-01-11 09:31:38 == I managed to get kdump to mount rootfs and perform its tasks by setting KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see http://pastebin.hursley.ibm.com/8239 I'm still investigating to figure out what is the reason behind this behavior. Thanks, -- maurosr == Comment: #23 - Mauricio Faria De Oliveira - 2017-01-11 11:56:40 == Mauro, (In reply to comment #22) > I managed to get kdump to mount rootfs and perform its tasks by setting > KDUMP_CMDLINE_APPEND="nr_cpus=4" parameter in /etc/default/kdump-tools see > http://pastebin.hursley.ibm.com/8239 > > I'm still investigating to figure out what is the reason behind this > behavior. > > Thanks, > > -- > maurosr That would smell like an out of memory condition that is alleviated with a smaller number of CPUs allowed for the kernel (so the amount of memory associated with per-CPU stuff is less in total). Per the bug description, the memory reserved for the crashkernel is 512MB: (In reply to comment #23) > - Configure kexec-tools to reserve sufficient RAM for the kdump kernel to > load (I use 512MB) in /etc/default/grub.d/kexec-tools.cfg That seems low for Power guests/systems. I think it theory is doesn't seem so, but the reality is that _for some reason(s)_ we require just too much memory to load and boot a kernel/initramfs (either on boot or kdump). When working w/ kdump and Ubuntu, I usually set the crashkernel allocated size right away to 4GB to avoid problems. Since this is a smaller sized guest, obviously we'd want to use less than t