[Kernel-packages] [Bug 1535088] Re: Linux Generic Kernel x -43
This may be the same thing we are seeing on OpenStack compute nodes since upgrade. Hard crash, no output to logs, local console occasionally shows a kernel panic ending with Fatal Exception in Interrupt, shutting down CPU's. We are seeing this reliably across 10 nodes which have been upgraded, and it doesn't happen on other nodes which are all running 3.13.0-62-generic. Have tried downgrading QEMU to 2.0, and OpenVswitch to 2.0.2-0ubuntu0.14.04.2, neither of which stop the problem happening. Hardware has also been stress tested and no memory or CPU issues. Still trying to find a reliable recreate, but seems to be connected with live migrations happening on other machines, where the node that crashes is hosting virtual instances on the same virtual networks on which the migration is happening. ** Attachment added: "Screen Shot 2016-01-21 at 05.15.02.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1535088/+attachment/4553994/+files/Screen%20Shot%202016-01-21%20at%2005.15.02.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1535088 Title: Linux Generic Kernel x -43 Status in linux package in Ubuntu: Invalid Bug description: kernel panic - not syncing : fatal exception in interrupt. shutting down cpus with NMI drm_kms_helper : panic occurred, switching back to text console. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-74-generic 3.13.0-74.118 ProcVersionSignature: Ubuntu 3.13.0-74.118-generic 3.13.11-ckt30 Uname: Linux 3.13.0-74-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.19 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC1: omniedge 2249 F pulseaudio /dev/snd/controlC0: omniedge 2249 F pulseaudio CurrentDesktop: Unity Date: Sun Jan 17 08:22:29 2016 EcryptfsInUse: Yes InstallationDate: Installed on 2015-01-29 (353 days ago) InstallationMedia: Ubuntu 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.2) MachineType: Dell Inc. Inspiron 3542 ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-74-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7 RelatedPackageVersions: linux-restricted-modules-3.13.0-74-generic N/A linux-backports-modules-3.13.0-74-generic N/A linux-firmware 1.127.19 SourcePackage: linux StagingDrivers: rts5139 UpgradeStatus: No upgrade log present (probably fresh install) WifiSyslog: dmi.bios.date: 11/14/2014 dmi.bios.vendor: Dell Inc. dmi.bios.version: A05 dmi.board.name: 02RD2H dmi.board.vendor: Dell Inc. dmi.board.version: A05 dmi.chassis.type: 8 dmi.chassis.vendor: Dell Inc. dmi.chassis.version: Not Specified dmi.modalias: dmi:bvnDellInc.:bvrA05:bd11/14/2014:svnDellInc.:pnInspiron3542:pvrNotSpecified:rvnDellInc.:rn02RD2H:rvrA05:cvnDellInc.:ct8:cvrNotSpecified: dmi.product.name: Inspiron 3542 dmi.product.version: Not Specified dmi.sys.vendor: Dell Inc. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1535088/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1383921] Re: kipmi0 process hangs with ipmitool
Actually forget that, just worked out they are in the extras package -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1383921 Title: kipmi0 process hangs with ipmitool Status in linux package in Ubuntu: In Progress Status in linux source package in Precise: Invalid Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: In Progress Status in linux source package in Vivid: In Progress Bug description: we're in the process of migrating from 12.04 to 14.04 and are noticing a problem apparently related to the kernel and ipmi. after an indeterminate period of regular ipmi queries (e.g., nagios checks using ipmitool) the kipmi0 process pegs a cpu to ~100% usage and further ipmitool commands hang. the former is not a huge problem as the process is niced and its cpu usage can be limited using the /sys/module/ipmi_si/parameters/kipmid_max_busy_us interface. however, ipmitool's hanging severely degrades hardware monitoring. this problem initially reared its head in our enviroment on a handful of 12.04 hosts on which hwe kernels were installed. of the dozen deployed 14.04 hosts, four have started displaying these symptoms in the past week. all of the 12.04+hwe hosts were eventually affected; i believe that given enough time all of the 14.04 hosts would be, as well. a reboot clears it up until its recurrence. red hat has a bug logged that appears to match this: https://bugzilla.redhat.com/show_bug.cgi?id=1090619 unfortunately the work-around and proposed fix (from a duplicate bug) are currently non- public. it does look like they were able to identify the ipmi: simplify locking patch from commit id f60adf42ad55405d1b17e9e5c33fdb63f1eb8861 as the culprit. i have just finished building a kernel from linux-source-3.13.0=3.13.0-37.64 w/this patch reversed and will deploy it to see if the problem is alleviated. thank you for your time and effort. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-37-generic 3.13.0-37.64 ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7 Uname: Linux 3.13.0-37-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 18 13:11 seq crw-rw 1 root audio 116, 33 Oct 18 13:11 timer AplayDevices: aplay: device_list:268: no soundcards found... ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 ArecordDevices: arecord: device_list:268: no soundcards found... AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Tue Oct 21 12:55:43 2014 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: Supermicro X8DT6 PciMultimedia: ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/usr/bin/zsh ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-37-generic root=/dev/md0 ro consoleblank=0 console=tty0 console=ttyS2,115200n8 nomdmonddf nomdmonisw bootdegraded=true RelatedPackageVersions: linux-restricted-modules-3.13.0-37-generic N/A linux-backports-modules-3.13.0-37-generic N/A linux-firmware 1.127.7 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 05/15/2012 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.0c dmi.board.asset.tag: 1234567890 dmi.board.name: X8DT6 dmi.board.vendor: Supermicro dmi.board.version: 1234567890 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 17 dmi.chassis.vendor: Supermicro dmi.chassis.version: 1234567890 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0c:bd05/15/2012:svnSupermicro:pnX8DT6:pvr1234567890:rvnSupermicro:rnX8DT6:rvr1234567890:cvnSupermicro:ct17:cvr1234567890: dmi.product.name: X8DT6 dmi.product.version: 1234567890 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1383921/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1383921] Re: kipmi0 process hangs with ipmitool
The kernel 3.13.0-41-generic which is in proposed does not appear to have any of the ipmi modules. Are we doing something wrong ? root@adam:/lib/modules# uname -r 3.13.0-41-generic root@adam:/lib/modules# modprobe ipmi_si modprobe: FATAL: Module ipmi_si not found. root@adam:/lib/modules# ipmitool lan print Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get Channel Info command failed Invalid channel: 0 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1383921 Title: kipmi0 process hangs with ipmitool Status in linux package in Ubuntu: In Progress Status in linux source package in Precise: Invalid Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: In Progress Status in linux source package in Vivid: In Progress Bug description: we're in the process of migrating from 12.04 to 14.04 and are noticing a problem apparently related to the kernel and ipmi. after an indeterminate period of regular ipmi queries (e.g., nagios checks using ipmitool) the kipmi0 process pegs a cpu to ~100% usage and further ipmitool commands hang. the former is not a huge problem as the process is niced and its cpu usage can be limited using the /sys/module/ipmi_si/parameters/kipmid_max_busy_us interface. however, ipmitool's hanging severely degrades hardware monitoring. this problem initially reared its head in our enviroment on a handful of 12.04 hosts on which hwe kernels were installed. of the dozen deployed 14.04 hosts, four have started displaying these symptoms in the past week. all of the 12.04+hwe hosts were eventually affected; i believe that given enough time all of the 14.04 hosts would be, as well. a reboot clears it up until its recurrence. red hat has a bug logged that appears to match this: https://bugzilla.redhat.com/show_bug.cgi?id=1090619 unfortunately the work-around and proposed fix (from a duplicate bug) are currently non- public. it does look like they were able to identify the ipmi: simplify locking patch from commit id f60adf42ad55405d1b17e9e5c33fdb63f1eb8861 as the culprit. i have just finished building a kernel from linux-source-3.13.0=3.13.0-37.64 w/this patch reversed and will deploy it to see if the problem is alleviated. thank you for your time and effort. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-37-generic 3.13.0-37.64 ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7 Uname: Linux 3.13.0-37-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 18 13:11 seq crw-rw 1 root audio 116, 33 Oct 18 13:11 timer AplayDevices: aplay: device_list:268: no soundcards found... ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 ArecordDevices: arecord: device_list:268: no soundcards found... AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Tue Oct 21 12:55:43 2014 IwConfig: Error: [Errno 2] No such file or
[Kernel-packages] [Bug 1383921] Re: kipmi0 process hangs with ipmitool
With the kernel 3.13.0-41-generic, I can confirm that we are now able to run warm and cold resets of the mc via ipmitool correctly. On previous kernel those commands would cause kipmi to use 100% cpu time and require the ipmi modules to be unloaded and reloaded before ipmitool would operate correctly. With this kernel once the mc is reset, ipmitool works properly. This would seem to fix the issue for us, but we're not the original bug reporter. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1383921 Title: kipmi0 process hangs with ipmitool Status in linux package in Ubuntu: In Progress Status in linux source package in Precise: Invalid Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: In Progress Status in linux source package in Vivid: In Progress Bug description: we're in the process of migrating from 12.04 to 14.04 and are noticing a problem apparently related to the kernel and ipmi. after an indeterminate period of regular ipmi queries (e.g., nagios checks using ipmitool) the kipmi0 process pegs a cpu to ~100% usage and further ipmitool commands hang. the former is not a huge problem as the process is niced and its cpu usage can be limited using the /sys/module/ipmi_si/parameters/kipmid_max_busy_us interface. however, ipmitool's hanging severely degrades hardware monitoring. this problem initially reared its head in our enviroment on a handful of 12.04 hosts on which hwe kernels were installed. of the dozen deployed 14.04 hosts, four have started displaying these symptoms in the past week. all of the 12.04+hwe hosts were eventually affected; i believe that given enough time all of the 14.04 hosts would be, as well. a reboot clears it up until its recurrence. red hat has a bug logged that appears to match this: https://bugzilla.redhat.com/show_bug.cgi?id=1090619 unfortunately the work-around and proposed fix (from a duplicate bug) are currently non- public. it does look like they were able to identify the ipmi: simplify locking patch from commit id f60adf42ad55405d1b17e9e5c33fdb63f1eb8861 as the culprit. i have just finished building a kernel from linux-source-3.13.0=3.13.0-37.64 w/this patch reversed and will deploy it to see if the problem is alleviated. thank you for your time and effort. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-37-generic 3.13.0-37.64 ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7 Uname: Linux 3.13.0-37-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 18 13:11 seq crw-rw 1 root audio 116, 33 Oct 18 13:11 timer AplayDevices: aplay: device_list:268: no soundcards found... ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 ArecordDevices: arecord: device_list:268: no soundcards found... AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Tue Oct 21 12:55:43 2014 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: Supermicro X8DT6 PciMultimedia: ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/usr/bin/zsh ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-37-generic root=/dev/md0 ro consoleblank=0 console=tty0 console=ttyS2,115200n8 nomdmonddf nomdmonisw bootdegraded=true RelatedPackageVersions: linux-restricted-modules-3.13.0-37-generic N/A linux-backports-modules-3.13.0-37-generic N/A linux-firmware 1.127.7 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 05/15/2012 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.0c dmi.board.asset.tag: 1234567890 dmi.board.name: X8DT6 dmi.board.vendor: Supermicro dmi.board.version: 1234567890 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 17 dmi.chassis.vendor: Supermicro dmi.chassis.version: 1234567890 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0c:bd05/15/2012:svnSupermicro:pnX8DT6:pvr1234567890:rvnSupermicro:rnX8DT6:rvr1234567890:cvnSupermicro:ct17:cvr1234567890: dmi.product.name: X8DT6 dmi.product.version: 1234567890 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1383921/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1383921] Re: kipmi0 process hangs with ipmitool
Is there any news on this issue ? we have many nodes in our Openstack cluster starting to suffer from this problem as we have a puppet fact which runs ipmitool lan print and so triggers the bug often. Could you also tell us what the workaround was for RHEL as we can't access the page on RedHat's site. The workaround we have found is to remove and re- add the ipmi_si module, but removing it takes a VERY long time ie. many many hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1383921 Title: kipmi0 process hangs with ipmitool Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Precise: Invalid Status in “linux” source package in Trusty: In Progress Status in “linux” source package in Utopic: In Progress Status in “linux” source package in Vivid: In Progress Bug description: we're in the process of migrating from 12.04 to 14.04 and are noticing a problem apparently related to the kernel and ipmi. after an indeterminate period of regular ipmi queries (e.g., nagios checks using ipmitool) the kipmi0 process pegs a cpu to ~100% usage and further ipmitool commands hang. the former is not a huge problem as the process is niced and its cpu usage can be limited using the /sys/module/ipmi_si/parameters/kipmid_max_busy_us interface. however, ipmitool's hanging severely degrades hardware monitoring. this problem initially reared its head in our enviroment on a handful of 12.04 hosts on which hwe kernels were installed. of the dozen deployed 14.04 hosts, four have started displaying these symptoms in the past week. all of the 12.04+hwe hosts were eventually affected; i believe that given enough time all of the 14.04 hosts would be, as well. a reboot clears it up until its recurrence. red hat has a bug logged that appears to match this: https://bugzilla.redhat.com/show_bug.cgi?id=1090619 unfortunately the work-around and proposed fix (from a duplicate bug) are currently non- public. it does look like they were able to identify the ipmi: simplify locking patch from commit id f60adf42ad55405d1b17e9e5c33fdb63f1eb8861 as the culprit. i have just finished building a kernel from linux-source-3.13.0=3.13.0-37.64 w/this patch reversed and will deploy it to see if the problem is alleviated. thank you for your time and effort. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-37-generic 3.13.0-37.64 ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7 Uname: Linux 3.13.0-37-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Oct 18 13:11 seq crw-rw 1 root audio 116, 33 Oct 18 13:11 timer AplayDevices: aplay: device_list:268: no soundcards found... ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 ArecordDevices: arecord: device_list:268: no soundcards found... AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Tue Oct 21 12:55:43 2014 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: Supermicro X8DT6 PciMultimedia: ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/usr/bin/zsh ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-37-generic root=/dev/md0 ro consoleblank=0 console=tty0 console=ttyS2,115200n8 nomdmonddf nomdmonisw bootdegraded=true RelatedPackageVersions: linux-restricted-modules-3.13.0-37-generic N/A linux-backports-modules-3.13.0-37-generic N/A linux-firmware 1.127.7 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 05/15/2012 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.0c dmi.board.asset.tag: 1234567890 dmi.board.name: X8DT6 dmi.board.vendor: Supermicro dmi.board.version: 1234567890 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 17 dmi.chassis.vendor: Supermicro dmi.chassis.version: 1234567890 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0c:bd05/15/2012:svnSupermicro:pnX8DT6:pvr1234567890:rvnSupermicro:rnX8DT6:rvr1234567890:cvnSupermicro:ct17:cvr1234567890: dmi.product.name: X8DT6 dmi.product.version: 1234567890 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1383921/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
