[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Changed in: ubuntu-power-systems Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Fix Released Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Released Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.04.3 --- makedumpfile (1:1.6.5-1ubuntu1~18.04.3) bionic; urgency=medium [ Guilherme G. Piccoli ] * Add kdump retry/delay mechanism when dumping over network (LP: #1681909) [ Thadeu Lima de Souza Cascardo ] * Use maxcpus instead of nr_cpus on ppc64el. (LP: #1828597) * ppc64: increase MAX_PHYSMEM_BITS to 2PB (LP: #1841288) [ Connor Kuehl ] * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860) -- Thadeu Lima de Souza Cascardo Wed, 09 Oct 2019 15:38:08 -0300 ** Changed in: makedumpfile (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Released Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1.3 --- makedumpfile (1:1.6.5-1ubuntu1.3) disco; urgency=medium [ Guilherme G. Piccoli ] * Add kdump retry/delay mechanism when dumping over network (LP: #1681909) [ Thadeu Lima de Souza Cascardo ] * Use maxcpus instead of nr_cpus on ppc64el. (LP: #1828597) * ppc64: increase MAX_PHYSMEM_BITS to 2PB (LP: #1841288) [ Connor Kuehl ] * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860) -- Thadeu Lima de Souza Cascardo Wed, 09 Oct 2019 15:33:57 -0300 ** Changed in: makedumpfile (Ubuntu Disco) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Released Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Released Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
There is some reported regressions in autopkgtests for ppc64el and s390x, I'm investigating. Cheers, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Retested with Bionic-proposed version (1:1.6.5-1ubuntu1~18.04.3) and Disco-proposed (1:1.6.5-1ubuntu1.3), both including the fix for this bug. All working fine. I've used the "droppkt" hack (see comment #24) to simulate the problem, and it's fixed in the -proposed version. Thanks, Guilherme ** Tags removed: verification-needed verification-needed-bionic verification-needed-disco ** Tags added: verification-done verification-done-bionic verification-done-disco -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Hello bugproxy, or anyone else affected, Accepted makedumpfile into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1.3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Disco) Status: In Progress => Fix Committed ** Tags removed: verification-done verification-done-disco ** Tags added: verification-needed verification-needed-disco ** Changed in: makedumpfile (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags removed: verification-done-bionic ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Just a quick update, the package got a respin without the offending patch, soon (after it got sponsored) it'll land into -proposed pocket. Thanks, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Even though the bionic and disco verifications were successful (thanks for verifying), these patches were bundled in a single submission with other patches (from other bugs) which could not be successfully verified. As a result, all patches have had to be removed. Next step is to re-upload new version of makedumpfile. ** Changed in: ubuntu-power-systems Status: Fix Committed => In Progress ** Changed in: makedumpfile (Ubuntu Bionic) Status: Fix Committed => In Progress ** Changed in: makedumpfile (Ubuntu Disco) Status: Fix Committed => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Tested with Bionic-proposed version (1:1.6.5-1ubuntu1~18.04.2) and Disco-proposed (1:1.6.5-1ubuntu1.1), both including the fix for this bug. All working fine. I've used the "droppkt" hack (see comment #24) in the Bionic version, to simulate the problem, and it's fixed in the -proposed version. Thanks, Guilherme ** Tags removed: verification-needed verification-needed-bionic verification-needed-disco ** Tags added: verification-done verification-done-bionic verification-done-disco -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Tags removed: sts-sponsor-slashd -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Changed in: ubuntu-power-systems Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Fix Committed Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Hello bugproxy, or anyone else affected, Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Hello bugproxy, or anyone else affected, Accepted makedumpfile into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: makedumpfile (Ubuntu Disco) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-disco -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: Fix Committed Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: Fix Committed Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Excellent news! Thanks Eric :-) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Hi Andrew Cloke, Yes, I'm currently sponsoring D/B for cascardo/gpicolli. Disco is already uploaded waiting for SRU team approval: https://launchpad.net/ubuntu/disco/+queue?queue_state=1_text=makedumpfile Bionic debdiff needs some rework before I do the final upload. - Eric -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Next step is to consider backport feasibility for disco and bionic. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
This bug was fixed in the package makedumpfile - 1:1.6.6-2ubuntu1 --- makedumpfile (1:1.6.6-2ubuntu1) eoan; urgency=medium [ Thadeu Lima de Souza Cascardo ] * Merge from Debian unstable. Remaining changes: - Bump amd64 crashkernel from 384M-:128M to 512M-:192M. * Add kdump retry/delay mechanism when dumping over network (LP: #1681909) * Allow proper reload of kdump after multiple hotplug events. (LP: #1828596) [ Connor Kuehl ] * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860) makedumpfile (1:1.6.6-2) unstable; urgency=medium [ Guilherme G. Piccoli ] * Add kdump retry/delay mechanism when dumping over network [ Thadeu Lima de Souza Cascardo ] * Use a different service for vmcore dump. * Use maxcpus instead of nr_cpus on ppc64el. * Reload kdump when CPU is brought online. * Allow proper reload of kdump after multiple hotplug events. makedumpfile (1:1.6.6-1) unstable; urgency=medium * Update to new upstream version 1.6.6. -- Thadeu Lima de Souza Cascardo Tue, 06 Aug 2019 12:18:15 -0300 ** Changed in: makedumpfile (Ubuntu Eoan) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Released Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Released Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Branch linked: lp:~gpiccoli/britney/hints-ubuntu -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Merge proposal to britney/hints-ubuntu submitted, to mark this test as "badtest" for ppc64el: https://code.launchpad.net/~gpiccoli/britney /hints-ubuntu/+merge/371479 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
After updating the package for eaon-proposed with the fix for ppc64 (thanks Eric!), I've manually tested that version and it's working fine, being able to collect the kernel crash dump. The version I've tested is: $ rmadison makedumpfile | grep eoan-proposed makedumpfile | 1:1.6.6-2ubuntu1 | eoan-proposed| source, amd64, arm64, armhf, i386, ppc64el, s390x Even with my positive test results, the autopkgtest is broken and keep failing for ppc64, according to: http://autopkgtest.ubuntu.com/packages/m/makedumpfile/eoan/ppc64el According to the above test, we can see 2 important things: a) The last time it succeeded was : "makedumpfile/1:1.6.5-1ubuntu2 2019-06-20". And we can see in fact the ppc64 test was Skipped, that's the reason it succeeded. b) Even the current released version for eoan, 1:1.6.5-1ubuntu2, is failing according to the autopkgtest that ran on: "makedumpfile/1:1.6.5-1ubuntu2 2019-07-25". This test was triggered due to makedumpfile being a reverse dependency of "file". Also, I've try to replicate the test in one local powerpc64 server, using autopkgtest. My command-line was: https://pastebin.ubuntu.com/p/y4KyRvJVmz/ I've switched 2 parameters, having 4 tests results: 1) With "--apt-pocket=proposed=src:makedumpfile" (to test -proposed version) and "--nova-reboot": http://paste.ubuntu.com/p/Z8dr6ssF2J/ 2) With "--apt-pocket=proposed=src:makedumpfile" (to test -proposed version) only http://paste.ubuntu.com/p/qxDRy5xPSZ/ 3) Without both parameters above (testing the released version) http://paste.ubuntu.com/p/cYdMrWHPsR/ 4) Only with "--nova-reboot": http://paste.ubuntu.com/p/xYC4KG9DZr/ In all cases I've failed, with "Broken Pipe" in a late part of the test. During the failure, I could even SSH into the testbed, so something clearly is wrong with the test. I'd like to ask hereby an exemption from this test, marking it as "badtest" in ppc64. Cascardo, do you agree? I think we could release this package before the Eoan freeze, which is sane (based on manual tests made by a colleague and I), we shouldn't block based on a test that was always skipped. We still plan to continue investigating the test failure, until we can fix it. Thanks, Guilherme -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Tags added: sts-sponsor-slashd -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
makedumpfile merge to "1:1.6.6-2ubuntu1" sponsored in Eoan. I appended the changelog to add the entry block[0] currently found in eoan-proposed that was missing to keep track of everything that has been done on the package: Since it was made by cascardo before 1:1.6.5-1ubuntu3 exist. Note: - I didn't want this to be a blocker for this upload due to many factors, but cascardo/gpicolli, can you guys have a look before the feature freeze[1] at this lintian report[2], it would be awesome. It's good to make the code more modern, but debian packaging too, especially when time permit like now (devel release). [0] makedumpfile (1:1.6.5-1ubuntu3) eoan; urgency=medium * debian/kdump-config.in: - Add kdump retry/delay mechanism when dumping over network. (LP: #1681909) -- gpicc...@canonical.com (Guilherme G. Piccoli) Thu, 04 Jul 2019 15:20:53 -0300 [1] - https://wiki.ubuntu.com/EoanErmine/ReleaseSchedule [2] - https://pastebin.canonical.com/p/dWYkNhwjCb/ - Eric -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Eric, thanks for looking into the testing failure.
I was discussing with Cascardo today, and he was able to spot quickly the
problem.
First, the reason "makedumpfile" test passed in ppc64el before was that the
test was skipped; Cascardo changed "makedumpfile" to be part of the called "big
packages"[0], in order the VMs used in the tests have more RAM. Now, the test
gets executed and fails if kernel version is >= 4.20.
The reason of the failure was that "makedumpfile" couldn't collect a
compressed dump, falling back to 'cp' - this led to the 'if' failure in
the test. The root cause is that kernel patch 4ffe713b7587 ("powerpc/mm:
Increase the max addressable memory to 2PB"), introduced in v4.20,
requires a counterpart in "makedumpfile", in the form of patch [1].
Without that, I was able to reproduce the problem locally:
$ makedumpfile -c -d 31 /proc/vmcore /var/crash/201908051539/dump-incomplete2
get_machdep_info_ppc64: Can't detect max_physmem_bits.
makedumpfile Failed.
When I've used kernel 4.18 in the same VM, I got:
$ makedumpfile -c -d 31 /proc/vmcore core.418
Copying data : [100.0 %] \ eta:
0s
The dumpfile is saved to core.418
The plan here according to Cascardo is to push makedumpfile 1.6.6 (already
containing the fix for the "physmem_bits" issue as well as my fix for this LP,
the retry/delay mechanism) to Eoan.
After that, in my understanding, we can move on with the SRU for Bionic/Disco.
Cheers,
Guilherme
[0] https://git.launchpad.net/~cascardo/autopkgtest-cloud/commit/?id=346b786925
[1] https://salsa.debian.org/debian/makedumpfile/commit/f349b51f
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to makedumpfile in Ubuntu.
https://bugs.launchpad.net/bugs/1681909
Title:
kdump is not captured in remote host when kdump over ssh is configured
Status in The Ubuntu-power-systems project:
In Progress
Status in makedumpfile package in Ubuntu:
Fix Committed
Status in makedumpfile source package in Xenial:
Won't Fix
Status in makedumpfile source package in Bionic:
In Progress
Status in makedumpfile source package in Cosmic:
Won't Fix
Status in makedumpfile source package in Disco:
In Progress
Status in makedumpfile source package in Eoan:
Fix Committed
Bug description:
[Impact]
* Kdump over network (like NFS mount or SSH dump) relies on network-
online target from systemd. Even so, there are some NICs that report
"Link Up" state but aren't ready to transmit packets. This is a
generally bad behavior that is credited probably to NIC firmware
delays, usually not fixable from drivers. Some adapters known to act
like this are bnx2x, tg3 and ixgbe.
* Kdump is a mechanism that may be a last resort to debug complex/hard
to reproduce issues, so it's interesting to increase its reliability /
resilience. We then propose here a solution/quirk to this issue on
network dump by adding a retry/delay mechanism; if it's a network
dump, kdump will retry some times and sleep between the attempts in
order to exclude the case of NICs that aren't ready yet but will soon
be able to transmit packets.
* Although first reported by IBM in PowerPC arch, the scope for this
issue is the NIC, and it was later reported in x86 arch too.
[Test case]
Usually it's difficult to naturally reproduce this issue in a deterministic
way, but we have an artificial test case on comment #24 of this LP.
Also, we have a report from this bug in which the user managed to reproduce
the problem consistently - it's fixed after testing our solution.
[Regression potential]
There's not a clear regression potential here since it's just a retry/delay
mechanism. Some potential problems may come from bad coding in the script.
The delay between attempts is only 3 sec per iteration, so it shouldn't block
the kdump progress for a high amount of time at once.
[Other information]
Salsa Debian commit:
https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Quick update: It seems to fail the same way with 1:1.6.5-1ubuntu2, so NOT introduced by this SRU via 1:1.6.5-1ubuntu3 We still have to test the autopkgtest locally on ppc64el arch and instrument/monitor the test to understand why no crash is found in /var/crash at the end of the test. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Quick update # excuses... page: makedumpfile (1:1.6.5-1ubuntu2 to 1:1.6.5-1ubuntu3) Maintainer: Louis Bouchard 0 days old autopkgtest for kpatch/0.5.0-0ubuntu2: amd64: Ignored failure autopkgtest for makedumpfile/1:1.6.5-1ubuntu3: amd64: Pass, arm64: Pass, armhf: Pass, i386: Pass, ppc64el: Regression â™» , s390x: Ignored failure Not considered # logs . makedumpfile: crash test: checking for crash file makedumpfile: ERROR: crash test: Found no compressed dumps . gpicolli and I are investigating the root cause. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Thanks Eric for sponsoring this LP! I'm marking Xenial as "Won't Fix" for now, since we had no issue reports and it'll require a slightly more complex backport than Bionic. We'll discuss about the SRU to Xenial in some time, specially if we have reports of this failure in that release. Cheers, Guilherme ** Changed in: makedumpfile (Ubuntu Xenial) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: Won't Fix Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Sponsored for 'Eoan'. We'll be able to start the SRU sponsoring as soon as it lands in -releases. Notes: * Patch lands in debian unstable ~2 weeks ago : https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f * Patch have been "Signed-off-by" by a member of the Ubuntu kernel team. ** Changed in: makedumpfile (Ubuntu Eoan) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: Fix Committed Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: Fix Committed Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Description changed: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. + + [Other information] + + Salsa Debian commit: + https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Marking Cosmic as 'Won't fix'. Ubuntu 18.10 (Cosmic Cuttlefish) End Of Life reached on July 18 2019. ** Changed in: makedumpfile (Ubuntu Cosmic) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: Won't Fix Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Patch added: "SRU for disco" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5278173/+files/makedumpfile_1.6.5-1ubuntu1.1_disco.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: In Progress Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
Fix for eoan at my ppa. ppa:cascardo/kdump2. Attaching SRU for disco and bionic. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: In Progress Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Patch added: "SRU for bionic" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5278174/+files/makedumpfile_1.6.5-1ubuntu1~18.04.2_bionic.debdiff -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: In Progress Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
The attachment "lp1681909_eoan.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: In Progress Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Changed in: ubuntu-power-systems Status: Confirmed => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: In Progress Status in makedumpfile package in Ubuntu: In Progress Status in makedumpfile source package in Xenial: In Progress Status in makedumpfile source package in Bionic: In Progress Status in makedumpfile source package in Cosmic: In Progress Status in makedumpfile source package in Disco: In Progress Status in makedumpfile source package in Eoan: In Progress Bug description: [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network- online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
This is the debdiff with the retry/delay mechanism, for Eoan. I've discussed with Cascardo and we agreed he will do the SRU to old releases (X/B/C/D) after applying some other SRUs he's working now. I'd like to thanks specially Hari, Murilo and Pavithra from IBM, that reported, worked and proposed a solution for this issue! ** Description changed: - == Comment: #0 - PAVITHRA R. PRAKASH - 2017-03-07 05:00:29 == - ---Problem Description--- + [Impact] - Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is - configured on firestone. + * Kdump over network (like NFS mount or SSH dump) relies on network- + online target from systemd. Even so, there are some NICs that report + "Link Up" state but aren't ready to transmit packets. This is a + generally bad behavior that is credited probably to NIC firmware delays, + usually not fixable from drivers. Some adapters known to act like this + are bnx2x, tg3 and ixgbe. - ---Steps to Reproduce--- + * Kdump is a mechanism that may be a last resort to debug complex/hard + to reproduce issues, so it's interesting to increase its reliability / + resilience. We then propose here a solution/quirk to this issue on + network dump by adding a retry/delay mechanism; if it's a network dump, + kdump will retry some times and sleep between the attempts in order to + exclude the case of NICs that aren't ready yet but will soon be able to + transmit packets. - 1. Configure kdump. - 2. Check whether kdump is operational using ?# kdump-config show?. - 3. Install ?kernel-debuginfo? and ?kernel-debuginfo-common? rpms. - 4. Setup password less ssh connection, generate rsa key. - # ssh-keygen -t rsa - 5. verify id_rsa and id_rsa.pub are created under /root/.ssh/ - 6. Edit /etc/default/kdump-tools and add below entries. - SSH="ubuntu@9.114.15.239" - SSH_KEY=/root/.ssh/id_rsa - 7. Propagate RSA key. - # kdump-config propagate - 8. Restart kdump service. - # kdump-config load - 9. Trigger Crash using below commands. - # echo "1" > /proc/sys/kernel/sysrq - # echo "c" > /proc/sysrq-trigger - 10. Verify dump is available in remote server in configured path. + * Although first reported by IBM in PowerPC arch, the scope for this + issue is the NIC, and it was later reported in x86 arch too. - Machine details - === + [Test case] - $ ipmitool -I lanplus -H 9.47.70.3 -U ADMIN -P admin sol activate + Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. + Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. - $ ssh ubuntu@9.47.70.29 + [Regression potential] - PW: shriya101 - - - Attaching logs - - == Comment: #1 - PAVITHRA R. PRAKASH - 2017-03-07 - 05:01:42 == - - - == Comment: #5 - PAVITHRA R. PRAKASH - 2017-03-07 23:19:46 == - Hi, - - Attaching the logs. - - Network info: - - root@ltc-firep3:~# hwinfo --network - 36: None 00.0: 10700 Loopback - [Created at net.126] - Unique ID: ZsBS.GQNx7L4uPNA - SysFS ID: /class/net/lo - Hardware Class: network interface - Model: "Loopback network interface" - Device File: lo - Link detected: yes - Config Status: cfg=new, avail=yes, need=no, active=unknown - - 37: None 00.0: 10701 Ethernet - [Created at net.126] - Unique ID: 2lHw.ndpeucax6V1 - Parent ID: mIXc.aXC4wIvegH8 - SysFS ID: /class/net/enP33p3s0f2 - SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.2 - Hardware Class: network interface - Model: "Ethernet network interface" - Driver: "tg3" - Driver Modules: "tg3" - Device File: enP33p3s0f2 - HW Address: 98:be:94:03:18:4a - Permanent HW Address: 98:be:94:03:18:4a - Link detected: no - Config Status: cfg=new, avail=yes, need=no, active=unknown - Attached to: #15 (Ethernet controller) - - 38: None 00.0: 10701 Ethernet - [Created at net.126] - Unique ID: 7Onn.ndpeucax6V1 - Parent ID: sx0U.aXC4wIvegH8 - SysFS ID: /class/net/enP33p3s0f0 - SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.0 - Hardware Class: network interface - Model: "Ethernet network interface" - Driver: "tg3" - Driver Modules: "tg3" - Device File: enP33p3s0f0 - HW Address: 98:be:94:03:18:48 - Permanent HW Address: 98:be:94:03:18:48 - Link detected: yes - Config Status: cfg=new, avail=yes, need=no, active=unknown - Attached to: #16 (Ethernet controller) - - 39: None 00.0: 10701 Ethernet - [Created at net.126] - Unique ID: VwX_.ndpeucax6V1 - Parent ID: DUng.aXC4wIvegH8 - SysFS ID: /class/net/enP33p3s0f3 - SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.3 - Hardware Class: network interface - Model: "Ethernet network interface" - Driver: "tg3" - Driver Modules: "tg3" - Device File: enP33p3s0f3 - HW
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
This is a mock reproducer of this issue by faking a 20s delay in virtio- net after its link is up. To enable that, user needs to build virtio-net with the hereby attached patch, and insert: "echo 1 > /sys/module/virtio_net/parameters/droppkt" in the /usr/sbin/kdump-config before the network dump procedure. It'll introduce a 20s delay in packet transmission, causing the issue reported in this LP to be reproduced. This patch was developed in Bionioc 4.15.x kernel. ** Patch added: "0001-NOT-UPSTREAM-virtio-net-Add-droppkt-param-to-force-d.patch" https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5275113/+files/0001-NOT-UPSTREAM-virtio-net-Add-droppkt-param-to-force-d.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Confirmed Status in makedumpfile package in Ubuntu: Confirmed Status in makedumpfile source package in Xenial: Confirmed Status in makedumpfile source package in Bionic: Confirmed Status in makedumpfile source package in Cosmic: Confirmed Status in makedumpfile source package in Disco: Confirmed Status in makedumpfile source package in Eoan: Confirmed Bug description: == Comment: #0 - PAVITHRA R. PRAKASH - 2017-03-07 05:00:29 == ---Problem Description--- Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is configured on firestone. ---Steps to Reproduce--- 1. Configure kdump. 2. Check whether kdump is operational using ?# kdump-config show?. 3. Install ?kernel-debuginfo? and ?kernel-debuginfo-common? rpms. 4. Setup password less ssh connection, generate rsa key. # ssh-keygen -t rsa 5. verify id_rsa and id_rsa.pub are created under /root/.ssh/ 6. Edit /etc/default/kdump-tools and add below entries. SSH="ubuntu@9.114.15.239" SSH_KEY=/root/.ssh/id_rsa 7. Propagate RSA key. # kdump-config propagate 8. Restart kdump service. # kdump-config load 9. Trigger Crash using below commands. # echo "1" > /proc/sys/kernel/sysrq # echo "c" > /proc/sysrq-trigger 10. Verify dump is available in remote server in configured path. Machine details === $ ipmitool -I lanplus -H 9.47.70.3 -U ADMIN -P admin sol activate $ ssh ubuntu@9.47.70.29 PW: shriya101 Attaching logs == Comment: #1 - PAVITHRA R. PRAKASH - 2017-03-07 05:01:42 == == Comment: #5 - PAVITHRA R. PRAKASH - 2017-03-07 23:19:46 == Hi, Attaching the logs. Network info: root@ltc-firep3:~# hwinfo --network 36: None 00.0: 10700 Loopback [Created at net.126] Unique ID: ZsBS.GQNx7L4uPNA SysFS ID: /class/net/lo Hardware Class: network interface Model: "Loopback network interface" Device File: lo Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown 37: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 2lHw.ndpeucax6V1 Parent ID: mIXc.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f2 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.2 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f2 HW Address: 98:be:94:03:18:4a Permanent HW Address: 98:be:94:03:18:4a Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #15 (Ethernet controller) 38: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 7Onn.ndpeucax6V1 Parent ID: sx0U.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f0 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.0 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f0 HW Address: 98:be:94:03:18:48 Permanent HW Address: 98:be:94:03:18:48 Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #16 (Ethernet controller) 39: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: VwX_.ndpeucax6V1 Parent ID: DUng.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f3 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.3 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f3 HW Address: 98:be:94:03:18:4b Permanent HW Address: 98:be:94:03:18:4b Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #25 (Ethernet controller) 40: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: bZ1s.ndpeucax6V1 Parent ID:
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
** Changed in: ubuntu-power-systems Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Confirmed Status in makedumpfile package in Ubuntu: Confirmed Status in makedumpfile source package in Xenial: Confirmed Status in makedumpfile source package in Bionic: Confirmed Status in makedumpfile source package in Cosmic: Confirmed Status in makedumpfile source package in Disco: Confirmed Status in makedumpfile source package in Eoan: Confirmed Bug description: == Comment: #0 - PAVITHRA R. PRAKASH - 2017-03-07 05:00:29 == ---Problem Description--- Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is configured on firestone. ---Steps to Reproduce--- 1. Configure kdump. 2. Check whether kdump is operational using ?# kdump-config show?. 3. Install ?kernel-debuginfo? and ?kernel-debuginfo-common? rpms. 4. Setup password less ssh connection, generate rsa key. # ssh-keygen -t rsa 5. verify id_rsa and id_rsa.pub are created under /root/.ssh/ 6. Edit /etc/default/kdump-tools and add below entries. SSH="ubuntu@9.114.15.239" SSH_KEY=/root/.ssh/id_rsa 7. Propagate RSA key. # kdump-config propagate 8. Restart kdump service. # kdump-config load 9. Trigger Crash using below commands. # echo "1" > /proc/sys/kernel/sysrq # echo "c" > /proc/sysrq-trigger 10. Verify dump is available in remote server in configured path. Machine details === $ ipmitool -I lanplus -H 9.47.70.3 -U ADMIN -P admin sol activate $ ssh ubuntu@9.47.70.29 PW: shriya101 Attaching logs == Comment: #1 - PAVITHRA R. PRAKASH - 2017-03-07 05:01:42 == == Comment: #5 - PAVITHRA R. PRAKASH - 2017-03-07 23:19:46 == Hi, Attaching the logs. Network info: root@ltc-firep3:~# hwinfo --network 36: None 00.0: 10700 Loopback [Created at net.126] Unique ID: ZsBS.GQNx7L4uPNA SysFS ID: /class/net/lo Hardware Class: network interface Model: "Loopback network interface" Device File: lo Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown 37: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 2lHw.ndpeucax6V1 Parent ID: mIXc.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f2 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.2 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f2 HW Address: 98:be:94:03:18:4a Permanent HW Address: 98:be:94:03:18:4a Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #15 (Ethernet controller) 38: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 7Onn.ndpeucax6V1 Parent ID: sx0U.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f0 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.0 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f0 HW Address: 98:be:94:03:18:48 Permanent HW Address: 98:be:94:03:18:48 Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #16 (Ethernet controller) 39: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: VwX_.ndpeucax6V1 Parent ID: DUng.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f3 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.3 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f3 HW Address: 98:be:94:03:18:4b Permanent HW Address: 98:be:94:03:18:4b Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #25 (Ethernet controller) 40: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: bZ1s.ndpeucax6V1 Parent ID: J7HY.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f1 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.1 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f1 HW Address: 98:be:94:03:18:49 Permanent HW Address: 98:be:94:03:18:49 Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #4 (Ethernet controller) root@ltc-firep3:~# Thanks, Pavithra == Comment: #6 - PAVITHRA R. PRAKASH - 2017-03-07 23:20:47 == == Comment: #7 -
[Kernel-packages] [Bug 1681909] Re: kdump is not captured in remote host when kdump over ssh is configured
It came to my attention that this issue still exists; an user report on x84-64 machine confirms that it's not only a ppc64 issue, it's a generic problem due to some NICs behavior; even with link up, they are really not ready to xmit packets. So, I could reproduce this by modifying virtio to mimic this scenario, and I'm working in a solution/quirk on kdump-tools (makedumpfile) package. Cheers, Guilherme ** Summary changed: - [FEAT 18.10] dump is not captured in remote host when kdump over ssh is configured on firestone. + kdump is not captured in remote host when kdump over ssh is configured ** No longer affects: makedumpfile (Ubuntu) ** Also affects: makedumpfile (Ubuntu) Importance: Undecided Status: New ** Changed in: makedumpfile (Ubuntu) Status: New => Confirmed ** Changed in: makedumpfile (Ubuntu) Importance: Undecided => Medium ** Changed in: makedumpfile (Ubuntu) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Also affects: makedumpfile (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Cosmic) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: makedumpfile (Ubuntu Eoan) Importance: Medium Assignee: Guilherme G. Piccoli (gpiccoli) Status: Confirmed ** Also affects: makedumpfile (Ubuntu Disco) Importance: Undecided Status: New ** Changed in: makedumpfile (Ubuntu Xenial) Status: New => Confirmed ** Changed in: makedumpfile (Ubuntu Bionic) Status: New => Confirmed ** Changed in: makedumpfile (Ubuntu Cosmic) Status: New => Confirmed ** Changed in: makedumpfile (Ubuntu Disco) Status: New => Confirmed ** Changed in: makedumpfile (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: makedumpfile (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: makedumpfile (Ubuntu Cosmic) Importance: Undecided => Medium ** Changed in: makedumpfile (Ubuntu Disco) Importance: Undecided => Medium ** Changed in: makedumpfile (Ubuntu Disco) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: makedumpfile (Ubuntu Cosmic) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: makedumpfile (Ubuntu Bionic) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: makedumpfile (Ubuntu Xenial) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Tags removed: targetmilestone-inin16043 ** Tags added: sts -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1681909 Title: kdump is not captured in remote host when kdump over ssh is configured Status in The Ubuntu-power-systems project: Incomplete Status in makedumpfile package in Ubuntu: Confirmed Status in makedumpfile source package in Xenial: Confirmed Status in makedumpfile source package in Bionic: Confirmed Status in makedumpfile source package in Cosmic: Confirmed Status in makedumpfile source package in Disco: Confirmed Status in makedumpfile source package in Eoan: Confirmed Bug description: == Comment: #0 - PAVITHRA R. PRAKASH - 2017-03-07 05:00:29 == ---Problem Description--- Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is configured on firestone. ---Steps to Reproduce--- 1. Configure kdump. 2. Check whether kdump is operational using ?# kdump-config show?. 3. Install ?kernel-debuginfo? and ?kernel-debuginfo-common? rpms. 4. Setup password less ssh connection, generate rsa key. # ssh-keygen -t rsa 5. verify id_rsa and id_rsa.pub are created under /root/.ssh/ 6. Edit /etc/default/kdump-tools and add below entries. SSH="ubuntu@9.114.15.239" SSH_KEY=/root/.ssh/id_rsa 7. Propagate RSA key. # kdump-config propagate 8. Restart kdump service. # kdump-config load 9. Trigger Crash using below commands. # echo "1" > /proc/sys/kernel/sysrq # echo "c" > /proc/sysrq-trigger 10. Verify dump is available in remote server in configured path. Machine details === $ ipmitool -I lanplus -H 9.47.70.3 -U ADMIN -P admin sol activate $ ssh ubuntu@9.47.70.29 PW: shriya101 Attaching logs == Comment: #1 - PAVITHRA R. PRAKASH - 2017-03-07 05:01:42 == == Comment: #5 - PAVITHRA R. PRAKASH - 2017-03-07 23:19:46 == Hi, Attaching the logs. Network info: root@ltc-firep3:~# hwinfo --network 36: None 00.0: 10700 Loopback [Created at net.126] Unique ID: ZsBS.GQNx7L4uPNA SysFS ID: /class/net/lo Hardware Class: network interface Model: "Loopback network interface" Device File: lo Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown 37: None 00.0: 10701 Ethernet [Created at net.126]
