[Kernel-packages] [Bug 1699772] Re: linux-image-4.10.0-24-generic, linux-image-4.8.0-56-generic, linux-image-4.4.0-81-generic, linux-image-3.13.0-121-generic Regression: many user-space apps crashing
This affects linux-aws 4.4.0-1020-aws as well. I ran into this issue on that kernel. https://github.com/collectd/collectd/issues/2321#issuecomment-311634825 Seems maybe fixed in 4.4.0-1022-aws ** Bug watch added: github.com/collectd/collectd/issues #2321 https://github.com/collectd/collectd/issues/2321 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1699772 Title: linux-image-4.10.0-24-generic, linux-image-4.8.0-56-generic, linux- image-4.4.0-81-generic, linux-image-3.13.0-121-generic Regression: many user-space apps crashing Status in LibreOffice: Won't Fix Status in commons-daemon package in Ubuntu: Confirmed Status in eclipse package in Ubuntu: Confirmed Status in imagej package in Ubuntu: Confirmed Status in libreoffice package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Status in octave package in Ubuntu: Confirmed Status in python-jpype package in Ubuntu: Confirmed Status in rustc package in Ubuntu: Confirmed Status in scilab package in Ubuntu: Confirmed Status in linux package in Debian: Confirmed Bug description: Distribution: Ubuntu 16.04 x64 (Flavour: KDE Neon User Edition 5.10) linux-image-4.4.0-81-generic appears to contain a regression, probably related to the CVE-2017-1000364 fix backport / patch. Using this kernel, the Oracle Java browser plugin always crashes during stack-related actions on initialization. This means, the plugin completely stopped working. It works perfectly fine in linux-image-4.4.0-79-generic (vurlerable to CVE-2017-1000364) as well as linux-image-4.11.6-041106-generic, which also contains a fix for CVE-2017-1000364. uname -a: > Linux Zweiblum 4.4.0-81-generic #104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux I tested Oracle Java 1.8 u131 as well as 1.6 u64 in Firefox 51.0.1 as well as Iceweasel / Firefox/3.5.16 in a chroot. Using linux-image-4.4.0-81-generic it crashes in all combinations while with both other kernels it works. I was not able to obtain any detailed crash information from Firefox 51.0.1, but Iceweasel 3.5.16 crashed completely, allowing me to obtain a stack trace which shows the relation to stack operations performed by the plugin, even without proper debug symbols: > (gdb) bt full > #0 0x7fa06d805307 in _expand_stack_to(unsigned char*) () from /opt/java-8-oracle/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #1 0x7fa06d8053ae in os::Linux::manually_expand_stack(JavaThread*, unsigned char*) () >from /opt/java-8-oracle/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #2 0x7fa06d80cf0b in JVM_handle_linux_signal () from /opt/java-8-oracle/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fa06d802e13 in signalHandler(int, siginfo*, void*) () from /opt/java-8-oracle/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 I first assumed a bug in the Java plugin, but it works fine in Linux 4.11.6. The crash will be triggered by any applet, for example the test applet at: * https://java.com/en/download/installed8.jsp I'm running the Ubuntu 16.04 based KDE Neon distribution which somehow apparently does not allow me to use apport to report this bug: > $ LANG= apport-cli linux-image-4.4.0-81-generic > > *** Collecting problem information > > The collected information can be sent to the developers to improve the > application. This might take a few minutes. > . > > *** Problem in linux-image-4.4.0-81-generic > > The problem cannot be reported: > > This is not an official KDE package. Please remove any third party package and try again. If someone can tell me how to get apport working for this package, I can use it to collect additional information, but (unfortunately?) the problem should be fairly easy to reproduce... To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1699772/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
> kthreadd invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 Yea - that 2nd order allocation failure is the exact same issue I was able to see (same GFP mask also) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-aws in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux-aws package in Ubuntu: New Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux source package in Xenial: Fix Released Status in linux-aws source package in Xenial: New Status in linux-raspi2 source package in Xenial: Confirmed Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I have seemingly solved this issue with linux-aws version 4.4.0-1016-aws at the very least. The specific issue I was seeing was 2nd order allocations failing when OOMKiller triggered. At the time I was thinking the issue was due to XFS and memory fragmentation with lots and lots of memory mapped files in Elasticsearch/Lucene. When we moved to EXT4 the rate of oomkiller firing dropped, but did not stop. We made the following 2 changes to sysctls which have effectively stopped higher order memory allocaitons from failing and oomkiller firing. Also these settings were used on i3.2xlarge hosts that have 60G of ram - your milage may vary. Also we do not run swap on our servers, so likely adding swap could have helped, but not an option for us. vm.min_free_kbytes = 100 # We set this to leave about 1G of ram available for the kernel in the hope that even if the memory was heavily fragmented there might still be enough memory for linux to grab a higher order memory allocation fast enough before oomkiller does things. vm.zone_reclaim_mode = 1 # our hope here was to get the kernel to get more aggressive in reclaiming memory -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-aws in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux-aws package in Ubuntu: New Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux source package in Xenial: Fix Released Status in linux-aws source package in Xenial: New Status in linux-raspi2 source package in Xenial: Confirmed Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@nicholas-hatch - what file system are your disks formatted as? I was able to stop the OOM's on my ES hosts by moving from XFS to EXT4. My belief is that there was a memory fragmentation issue with ES and many small files on XFS formatted volumes. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux source package in Xenial: Fix Released Status in linux-raspi2 source package in Xenial: Confirmed Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
I've tried setting vm.vfs_cache_pressure = 300 per the top post and still seeing regular (daily) oom's on 4.4.0-66-generic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We have been seeing this issue recently as well. We are running 4.4.0-66-generic #87-Ubuntu - I can attempt to downgrade to 4.4.0-57 but its a large cluster with a lot of data so it may take some time. Attached a kern.log from this most recent oom. ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4841016/+files/kern.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1668129] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827357/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Feb 27 02:12 seq crw-rw 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
[Kernel-packages] [Bug 1668129] Lspci.txt
apport information ** Attachment added: "Lspci.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827355/+files/Lspci.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Feb 27 02:12 seq crw-rw 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
[Kernel-packages] [Bug 1668129] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827359/+files/UdevDb.txt ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Feb 27 02:12 seq crw-rw 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list:
[Kernel-packages] [Bug 1668129] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827358/+files/ProcModules.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Feb 27 02:12 seq crw-rw 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
[Kernel-packages] [Bug 1668129] ProcCpuinfo.txt
apport information ** Attachment added: "ProcCpuinfo.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827356/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Feb 27 02:12 seq crw-rw 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
[Kernel-packages] [Bug 1668129] Re: Amazon I3 Instance Buffer I/O error on dev nvme0n1
apport information ** Tags added: apport-collected ec2-images xenial ** Description changed: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 - Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 - 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux + Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Feb 27 02:12 seq + crw-rw 1 root audio 116, 33 Feb 27 02:12 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.20.1-0ubuntu2.5 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: N/A + DistroRelease: Ubuntu 16.04 + Ec2AMI: ami-bc62b2aa + Ec2AMIManifest: (unknown) + Ec2AvailabilityZone: us-east-1d + Ec2InstanceType: i3.2xlarge + Ec2Kernel: unavailable + Ec2Ramdisk: unavailable + IwConfig: Error: [Errno 2] No such file or directory + JournalErrors: + Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. +Users in the 'systemd-journal' group can see all messages. Pass -q to +turn off this notice. + No journal files were opened due to insufficient permissions. + Lsusb: Error: command ['lsusb'] failed with exit code 1: + MachineType: Xen HVM domU + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen-256color + PATH=(custom, no user) + XDG_RUNTIME_DIR= + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: + + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 + ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 + RelatedPackageVersions: + linux-restricted-modules-4.4.0-64-generic N/A + linux-backports-modules-4.4.0-64-generic N/A + linux-firmwareN/A + RfKill: Error: [Errno 2] No such file or directory + Tags: xenial ec2-images + Uname: Linux 4.4.0-64-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + WifiSyslog: + + _MarkForUpload: True + dmi.bios.date: 12/12/2016 + dmi.bios.vendor: Xen + dmi.bios.version: 4.2.amazon + dmi.chassis.type: 1 + dmi.chassis.vendor: Xen + dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: + dmi.product.name: HVM domU + dmi.product.version: 4.2.amazon + dmi.sys.vendor: Xen ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1668129/+attachment/4827354/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: Confirmed Bug description:
[Kernel-packages] [Bug 1668129] [NEW] Amazon I3 Instance Buffer I/O error on dev nvme0n1
Public bug reported: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Attachment added: "lspci-vnvn.log" https://bugs.launchpad.net/bugs/1668129/+attachment/4827353/+files/lspci-vnvn.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1668129 Title: Amazon I3 Instance Buffer I/O error on dev nvme0n1 Status in linux package in Ubuntu: New Bug description: On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1450442] Re: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [ffffffff810fb39b] ? audit_compare_dname_path+0x2b/0xa0
I'm still able to recreate this issue with kernel version 3.13.0-52-generic #85-Ubuntu SMP Wed Apr 29 16:44:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux It looks like a different set of audit rules causes the same issue. To replicate: Install 3.13.0-52-generic kernel apt-get install auditd in /etc/audit/audit.rules --- -D -b 5000 -f 0 -r 15000 -a exit,always -F arch=b64 -S execve -S exit -S exit_group -S fork -S clone -S vfork -S accept -S accept4 -S connect -S bind -S listen --- restart auditd below stacktrace happens. Stacktrace: [ 186.897309] BUG: unable to handle kernel NULL pointer dereference at 0690 [ 186.897322] IP: [8136cbb0] strlen+0x0/0x30 [ 186.897331] PGD 0 [ 186.897334] Oops: [#1] SMP [ 186.897339] Modules linked in: dm_crypt crct10dif_pclmul crc32_pclmul ghash_clmulni_intel isofs aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd [ 186.897357] CPU: 0 PID: 2206 Comm: sudo Not tainted 3.13.0-52-generic #85-Ubuntu [ 186.897363] task: 880003286000 ti: 880002a04000 task.ti: 880002a04000 [ 186.897368] RIP: e030:[8136cbb0] [8136cbb0] strlen+0x0/0x30 [ 186.897375] RSP: e02b:880002a05df0 EFLAGS: 00010286 [ 186.897379] RAX: 880002a05d40 RBX: 0690 RCX: [ 186.897382] RDX: 0036 RSI: 0690 RDI: 0690 [ 186.897385] RBP: 880002a05e08 R08: R09: fffe [ 186.897389] R10: R11: 880002a05c06 R12: 8801d298f340 [ 186.897393] R13: R14: 8801d0fa2000 R15: [ 186.897401] FS: 7f4a94370840() GS:8801dee0() knlGS: [ 186.897408] CS: e033 DS: ES: CR0: 8005003b [ 186.897412] CR2: 0690 CR3: 031f5000 CR4: 2660 [ 186.897418] Stack: [ 186.897420] 810f7fda 8801d298f340 8801d0fa2060 880002a05e78 [ 186.897425] 810f9581 8172a480 81c55740 880002a05e60 [ 186.897430] 8172a480 880002a05ef0 880002a05e60 810f6b93 [ 186.897435] Call Trace: [ 186.897441] [810f7fda] ? audit_log_untrustedstring+0x1a/0x30 [ 186.897445] [810f9581] audit_log_name+0x281/0x320 [ 186.897451] [8172a480] ? _raw_spin_unlock_irqrestore+0x20/0x40 [ 186.897455] [8172a480] ? _raw_spin_unlock_irqrestore+0x20/0x40 [ 186.897459] [810f6b93] ? audit_buffer_free+0x73/0xa0 [ 186.897463] [810fbe37] audit_log_exit+0x3d7/0xb90 [ 186.897467] [810fe5bf] __audit_syscall_exit+0x27f/0x2e0 [ 186.897472] [81733224] sysret_audit+0x17/0x21 [ 186.897474] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 186.897508] RIP [8136cbb0] strlen+0x0/0x30 [ 186.897511] RSP 880002a05df0 [ 186.897513] CR2: 0690 [ 186.897516] ---[ end trace 2626030fc35ecb54 ]--- -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1450442 Title: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: [Impact] Ubuntu VMWare instances running 3.13.0-51 will crash with the following backtrace: [ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) [ 12.357886] IP: [8136cb80] strlen+0x0/0x30 [ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0 [ 12.359034] Oops: [#1] SMP [ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase [ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu [ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 [ 12.367276] task: 880230fc3000 ti: 8802308c4000 task.ti: 8802308c4000 [ 12.368159] RIP: 0010:[8136cb80]
[Kernel-packages] [Bug 1450442] Re: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [ffffffff810fb39b] ? audit_compare_dname_path+0x2b/0xa0
Ah - crap - sorry about that. You are right. Thanks! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1450442 Title: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: [Impact] Ubuntu VMWare instances running 3.13.0-51 will crash with the following backtrace: [ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) [ 12.357886] IP: [8136cb80] strlen+0x0/0x30 [ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0 [ 12.359034] Oops: [#1] SMP [ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase [ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu [ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 [ 12.367276] task: 880230fc3000 ti: 8802308c4000 task.ti: 8802308c4000 [ 12.368159] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 [ 12.369073] RSP: 0018:8802308c5d60 EFLAGS: 00010212 [ 12.369963] RAX: 000d RBX: 000d RCX: 2df0 [ 12.370973] RDX: 0012 RSI: RDI: [ 12.372005] RBP: 8802308c5d90 R08: 8800b9218648 R09: 8802308c5d60 [ 12.372988] R10: 0002 R11: 88023082e180 R12: 0012 [ 12.373901] R13: R14: 880231f1b3f8 R15: 8800b9218460 [ 12.374827] FS: 7f196f84c740() GS:88023fd0() knlGS: [ 12.375752] CS: 0010 DS: ES: CR0: 80050033 [ 12.376667] CR2: CR3: 000230872000 CR4: 000407e0 [ 12.377684] Stack: [ 12.378612] 810fb39b 0004 88022ff74838 [ 12.379559] 8800b9218400 8800b9218460 8802308c5df8 810fdb36 [ 12.380516] 811d56e0 00042ff74838 880231f1b3c0 88022febecf8 [ 12.381506] Call Trace: [ 12.382630] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 [ 12.383784] [810fdb36] __audit_inode_child+0xb6/0x330 [ 12.384912] [811d56e0] ? d_instantiate+0x50/0x70 [ 12.386013] [811ca060] vfs_mknod+0x110/0x160 [ 12.387145] [816bf475] unix_bind+0x2a5/0x360 [ 12.388207] [810ff142] ? __audit_sockaddr+0x42/0x80 [ 12.389250] [8160d4c0] SYSC_bind+0xe0/0x120 [ 12.390297] [8172e9fa] ? do_page_fault+0x1a/0x70 [ 12.391303] [8160e4de] SyS_bind+0xe/0x10 [ 12.392426] [817330bd] system_call_fastpath+0x1a/0x1f [ 12.393581] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 12.396831] RIP [8136cb80] strlen+0x0/0x30 [ 12.397812] RSP 8802308c5d60 [ 12.398769] CR2: [ 12.399743] ---[ end trace 2c5a33d31a03347e ]--- We've also seen this on our precise machines that are running the backported trusty kernel. When reverting to kernel 3.13.0-49 this no longer occurs. [Test Case] 1) Run an Ubuntu VMWare instance with the affected kernel. apt-get install auditd echo -w /etc/test /etc/audit/audit.rules /etc/init.d/auditd restart apt-get install linux-headers-3.13.0-51 linux-headers-3.13.0-51-generic linux-image-3.13.0-51-generic reboot attempt to login or ssh into the host - you'll get a similar stacktrace. [Fix] commit fcf22d8267ad2601fe9b6c549d1be96401c23e0b upstream -- uname -a: Linux search-2 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux cat /proc/version_signature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450442/+subscriptions -- Mailing list:
[Kernel-packages] [Bug 1450643] Re: Installing linux-image-3.13.0-51-generic with auditd causes inability to ssh into the system
*** This bug is a duplicate of bug 1450442 *** https://bugs.launchpad.net/bugs/1450442 Additional way to replicate Setup a vanilla ubuntu 14.04 system apt-get install linux-headers-3.13.0-51 linux-headers-3.13.0-51-generic linux-image-3.13.0-51-generic reboot now running 3.13.0-51 login apt-get install auditd echo -w /etc/test /etc/audit/audit.rules /etc/init.d/auditd restart logout ssh to the system failure - and below stacktrace Thru doing more testing - depending on the order you take depends on what fails. But the main issue is that you will be unable to ssh to the machine and get the below stacktrace Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.897427] BUG: unable to handle kernel NULL pointer dereference at 0023 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.897696] IP: [8136cb80] strlen+0x0/0x30 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.897867] PGD 0 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.897977] Oops: [#3] SMP Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.898150] Modules linked in: vmw_vsock_vmci_transport vsock dm_crypt ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd psmouse serio_raw nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache vmw_vmci parport_pc lp parport vmw_pvscsi vmxnet3 mptspi e1000 mptscsih mptbase floppy Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.899575] CPU: 0 PID: 1646 Comm: sshd Tainted: G D 3.13.0-51-generic #84-Ubuntu Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.899790] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.900023] task: 8800078c6000 ti: 8800144e8000 task.ti: 8800144e8000 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.900241] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.900489] RSP: 0018:8800144e9d60 EFLAGS: 00010216 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.900598] RAX: 000a RBX: 000a RCX: 001dfcd3 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.900752] RDX: 0014 RSI: 0023 RDI: 0023 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.901749] RBP: 8800144e9d90 R08: 880015789248 R09: 880017c197e0 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.902249] R10: 81288580 R11: ea4e0b40 R12: 0014 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.902512] R13: 0023 R14: 88000b07cdb8 R15: 880015789060 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.902738] FS: 7f2117494840() GS:880017c0() knlGS: Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.902976] CS: 0010 DS: ES: CR0: 80050033 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.903142] CR2: 0023 CR3: 0342e000 CR4: 001407f0 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.903374] Stack: Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.903495] 810fb39b 0004 88001318c488 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.903834] 880015789000 880015789060 8800144e9df8 810fdb36 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.904170] 08e6 00040023 88000b07cd80 88001318cc38 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.904505] Call Trace: Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.904636] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.904795] [810fdb36] __audit_inode_child+0xb6/0x330 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.904958] [811ca060] vfs_mknod+0x110/0x160 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905106] [816bf475] unix_bind+0x2a5/0x360 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905251] [810ff142] ? __audit_sockaddr+0x42/0x80 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905404] [8160d4c0] SYSC_bind+0xe0/0x120 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905548] [8160e4de] SyS_bind+0xe/0x10 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905691] [817330bd] system_call_fastpath+0x1a/0x1f Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.905839] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.907914] RIP [8136cb80] strlen+0x0/0x30 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.910466] RSP 8800144e9d60 Apr 30 22:06:22 default-ubuntu-1404 kernel: [ 253.910568] CR2: 0023 Apr 30
[Kernel-packages] [Bug 1450442] Re: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [ffffffff810fb39b] ? audit_compare_dname_path+0x2b/0xa0
I've tested the build from http://people.canonical.com/~arges/lp1450442/ - and i'm no longer able to replicate this issue. This looks like it works for me. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1450442 Title: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 Status in linux package in Ubuntu: Confirmed Status in linux source package in Trusty: In Progress Status in linux source package in Utopic: In Progress Bug description: [Impact] Ubuntu VMWare instances running 3.13.0-51 will crash with the following backtrace: [ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) [ 12.357886] IP: [8136cb80] strlen+0x0/0x30 [ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0 [ 12.359034] Oops: [#1] SMP [ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase [ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu [ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 [ 12.367276] task: 880230fc3000 ti: 8802308c4000 task.ti: 8802308c4000 [ 12.368159] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 [ 12.369073] RSP: 0018:8802308c5d60 EFLAGS: 00010212 [ 12.369963] RAX: 000d RBX: 000d RCX: 2df0 [ 12.370973] RDX: 0012 RSI: RDI: [ 12.372005] RBP: 8802308c5d90 R08: 8800b9218648 R09: 8802308c5d60 [ 12.372988] R10: 0002 R11: 88023082e180 R12: 0012 [ 12.373901] R13: R14: 880231f1b3f8 R15: 8800b9218460 [ 12.374827] FS: 7f196f84c740() GS:88023fd0() knlGS: [ 12.375752] CS: 0010 DS: ES: CR0: 80050033 [ 12.376667] CR2: CR3: 000230872000 CR4: 000407e0 [ 12.377684] Stack: [ 12.378612] 810fb39b 0004 88022ff74838 [ 12.379559] 8800b9218400 8800b9218460 8802308c5df8 810fdb36 [ 12.380516] 811d56e0 00042ff74838 880231f1b3c0 88022febecf8 [ 12.381506] Call Trace: [ 12.382630] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 [ 12.383784] [810fdb36] __audit_inode_child+0xb6/0x330 [ 12.384912] [811d56e0] ? d_instantiate+0x50/0x70 [ 12.386013] [811ca060] vfs_mknod+0x110/0x160 [ 12.387145] [816bf475] unix_bind+0x2a5/0x360 [ 12.388207] [810ff142] ? __audit_sockaddr+0x42/0x80 [ 12.389250] [8160d4c0] SYSC_bind+0xe0/0x120 [ 12.390297] [8172e9fa] ? do_page_fault+0x1a/0x70 [ 12.391303] [8160e4de] SyS_bind+0xe/0x10 [ 12.392426] [817330bd] system_call_fastpath+0x1a/0x1f [ 12.393581] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 12.396831] RIP [8136cb80] strlen+0x0/0x30 [ 12.397812] RSP 8802308c5d60 [ 12.398769] CR2: [ 12.399743] ---[ end trace 2c5a33d31a03347e ]--- We've also seen this on our precise machines that are running the backported trusty kernel. When reverting to kernel 3.13.0-49 this no longer occurs. [Test Case] Run an Ubuntu VMWare instance with the affected kernel. See crash on boot. [Fix] commit fcf22d8267ad2601fe9b6c549d1be96401c23e0b upstream -- uname -a: Linux search-2 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux cat /proc/version_signature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450442/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1450643] Re: Installing linux-image-3.13.0-51-generic with auditd causes unrecoverable system state
*** This bug is a duplicate of bug 1450442 *** https://bugs.launchpad.net/bugs/1450442 While i'm not sure exactly - this seems to be the only commit in 3.13.0-51 around the audit area. https://lkml.org/lkml/2014/9/5/66 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-lts-trusty in Ubuntu. https://bugs.launchpad.net/bugs/1450643 Title: Installing linux-image-3.13.0-51-generic with auditd causes inability to ssh into the system Status in linux-lts-trusty package in Ubuntu: New Bug description: When installing linux kernel 3.13.0-51 onto a system that has auditd installed with at least one rule enabled, the following stacktrace occurs when various processes occur (pretty much anything by root). [ 41.813593] type=1305 audit(1430428709.300:28): audit_pid=0 old=880 auid=4294967295 ses=4294967295 res=1 [ 41.830177] type=1305 audit(1430428709.316:29): audit_enabled=1 old=1 auid=900 ses=1 res=1 [ 41.830184] type=1305 audit(1430428709.316:30): audit_pid=1238 old=0 auid=900 ses=1 res=1 [ 51.377064] BUG: unable to handle kernel NULL pointer dereference at 0023 [ 51.377243] IP: [8136cb80] strlen+0x0/0x30 [ 51.377312] PGD 0 [ 51.377358] Oops: [#1] SMP [ 51.377428] Modules linked in: vmw_vsock_vmci_transport vsock dm_crypt ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd psmouse serio_raw nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache vmw_vmci parport_pc lp parport vmw_pvscsi vmxnet3 e1000 mptspi mptscsih mptbase floppy [ 51.378232] CPU: 0 PID: 1281 Comm: sshd Not tainted 3.13.0-51-generic #84-Ubuntu [ 51.378285] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014 [ 51.378355] task: 880015979800 ti: 8800158f6000 task.ti: 8800158f6000 [ 51.378429] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 [ 51.378496] RSP: 0018:8800158f7d60 EFLAGS: 00010216 [ 51.378537] RAX: 000a RBX: 000a RCX: 001e0e67 [ 51.378587] RDX: 0014 RSI: 0023 RDI: 0023 [ 51.378637] RBP: 8800158f7d90 R08: 88001549c248 R09: 880017c197e0 [ 51.378736] R10: 81288580 R11: ea020ac0 R12: 0014 [ 51.378823] R13: 0023 R14: 88000f1661b8 R15: 88001549c060 [ 51.378910] FS: 7fee7e146840() GS:880017c0() knlGS: [ 51.379038] CS: 0010 DS: ES: CR0: 80050033 [ 51.379118] CR2: 0023 CR3: 175f4000 CR4: 001407f0 [ 51.379238] Stack: [ 51.379299] 810fb39b 0004 88000f1633e8 [ 51.379523] 88001549c000 88001549c060 8800158f7df8 810fdb36 [ 51.379827] 08e6 00040023 88000f166180 88000f1637c0 [ 51.380015] Call Trace: [ 51.380085] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 [ 51.380170] [810fdb36] __audit_inode_child+0xb6/0x330 [ 51.380254] [811ca060] vfs_mknod+0x110/0x160 [ 51.380350] [816bf475] unix_bind+0x2a5/0x360 [ 51.380449] [810ff142] ? __audit_sockaddr+0x42/0x80 [ 51.380538] [8160d4c0] SYSC_bind+0xe0/0x120 [ 51.380617] [8160e4de] SyS_bind+0xe/0x10 [ 51.380777] [817330bd] system_call_fastpath+0x1a/0x1f [ 51.380943] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 51.382310] RIP [8136cb80] strlen+0x0/0x30 [ 51.382432] RSP 8800158f7d60 [ 51.382518] CR2: 0023 [ 51.382654] ---[ end trace b5e4e75ed1093ea0 ]--- To Replicate: apt-get install auditd echo -w /etc/test /etc/audit/audit.rules /etc/init.d/auditd restart apt-get install linux-headers-3.13.0-51 linux-headers-3.13.0-51-generic linux-image-3.13.0-51-generic reboot attempt to login or ssh into the host - you'll get a similar stacktrace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1450643/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1450442] Re: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [ffffffff810fb39b] ? audit_compare_dname_path+0x2b/0xa0
It looks like this might be related? https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1450643 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1450442 Title: Kernel Oops - unable to handle kernel NULL pointer dereference at (null); Call Trace: [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 Status in linux package in Ubuntu: Confirmed Status in linux source package in Trusty: In Progress Status in linux source package in Utopic: In Progress Bug description: [Impact] Ubuntu VMWare instances running 3.13.0-51 will crash with the following backtrace: [ 12.357276] BUG: unable to handle kernel NULL pointer dereference at (null) [ 12.357886] IP: [8136cb80] strlen+0x0/0x30 [ 12.358457] PGD 230fe9067 PUD 230d5c067 PMD 0 [ 12.359034] Oops: [#1] SMP [ 12.359590] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack crct10dif_pclmul iptable_filter crc32_pclmul ip_tables ghash_clmulni_intel aesni_intel aes_x86_64 ppdev lrw x_tables gf128mul vmwgfx glue_helper ablk_helper cryptd ttm drm vmw_balloon serio_raw shpchp parport_pc lp i2c_piix4 parport mac_hid vmw_vmci psmouse mptspi vmw_pvscsi e1000 mptscsih floppy vmxnet3 mptbase [ 12.364773] CPU: 2 PID: 1718 Comm: fail2ban-server Not tainted 3.13.0-51-generic #84-Ubuntu [ 12.365587] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 [ 12.367276] task: 880230fc3000 ti: 8802308c4000 task.ti: 8802308c4000 [ 12.368159] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 [ 12.369073] RSP: 0018:8802308c5d60 EFLAGS: 00010212 [ 12.369963] RAX: 000d RBX: 000d RCX: 2df0 [ 12.370973] RDX: 0012 RSI: RDI: [ 12.372005] RBP: 8802308c5d90 R08: 8800b9218648 R09: 8802308c5d60 [ 12.372988] R10: 0002 R11: 88023082e180 R12: 0012 [ 12.373901] R13: R14: 880231f1b3f8 R15: 8800b9218460 [ 12.374827] FS: 7f196f84c740() GS:88023fd0() knlGS: [ 12.375752] CS: 0010 DS: ES: CR0: 80050033 [ 12.376667] CR2: CR3: 000230872000 CR4: 000407e0 [ 12.377684] Stack: [ 12.378612] 810fb39b 0004 88022ff74838 [ 12.379559] 8800b9218400 8800b9218460 8802308c5df8 810fdb36 [ 12.380516] 811d56e0 00042ff74838 880231f1b3c0 88022febecf8 [ 12.381506] Call Trace: [ 12.382630] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 [ 12.383784] [810fdb36] __audit_inode_child+0xb6/0x330 [ 12.384912] [811d56e0] ? d_instantiate+0x50/0x70 [ 12.386013] [811ca060] vfs_mknod+0x110/0x160 [ 12.387145] [816bf475] unix_bind+0x2a5/0x360 [ 12.388207] [810ff142] ? __audit_sockaddr+0x42/0x80 [ 12.389250] [8160d4c0] SYSC_bind+0xe0/0x120 [ 12.390297] [8172e9fa] ? do_page_fault+0x1a/0x70 [ 12.391303] [8160e4de] SyS_bind+0xe/0x10 [ 12.392426] [817330bd] system_call_fastpath+0x1a/0x1f [ 12.393581] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 12.396831] RIP [8136cb80] strlen+0x0/0x30 [ 12.397812] RSP 8802308c5d60 [ 12.398769] CR2: [ 12.399743] ---[ end trace 2c5a33d31a03347e ]--- We've also seen this on our precise machines that are running the backported trusty kernel. When reverting to kernel 3.13.0-49 this no longer occurs. [Test Case] Run an Ubuntu VMWare instance with the affected kernel. See crash on boot. [Fix] commit fcf22d8267ad2601fe9b6c549d1be96401c23e0b upstream -- uname -a: Linux search-2 3.13.0-51-generic #84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux cat /proc/version_signature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450442/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1450643] [NEW] Installing linux-image-3.13.0-51-generic with auditd causes unrecoverable system state
Public bug reported: When installing linux kernel 3.13.0-51 onto a system that has auditd installed with at least one rule enabled, the following stacktrace occurs when various processes occur (pretty much anything by root). [ 41.813593] type=1305 audit(1430428709.300:28): audit_pid=0 old=880 auid=4294967295 ses=4294967295 res=1 [ 41.830177] type=1305 audit(1430428709.316:29): audit_enabled=1 old=1 auid=900 ses=1 res=1 [ 41.830184] type=1305 audit(1430428709.316:30): audit_pid=1238 old=0 auid=900 ses=1 res=1 [ 51.377064] BUG: unable to handle kernel NULL pointer dereference at 0023 [ 51.377243] IP: [8136cb80] strlen+0x0/0x30 [ 51.377312] PGD 0 [ 51.377358] Oops: [#1] SMP [ 51.377428] Modules linked in: vmw_vsock_vmci_transport vsock dm_crypt ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd psmouse serio_raw nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache vmw_vmci parport_pc lp parport vmw_pvscsi vmxnet3 e1000 mptspi mptscsih mptbase floppy [ 51.378232] CPU: 0 PID: 1281 Comm: sshd Not tainted 3.13.0-51-generic #84-Ubuntu [ 51.378285] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014 [ 51.378355] task: 880015979800 ti: 8800158f6000 task.ti: 8800158f6000 [ 51.378429] RIP: 0010:[8136cb80] [8136cb80] strlen+0x0/0x30 [ 51.378496] RSP: 0018:8800158f7d60 EFLAGS: 00010216 [ 51.378537] RAX: 000a RBX: 000a RCX: 001e0e67 [ 51.378587] RDX: 0014 RSI: 0023 RDI: 0023 [ 51.378637] RBP: 8800158f7d90 R08: 88001549c248 R09: 880017c197e0 [ 51.378736] R10: 81288580 R11: ea020ac0 R12: 0014 [ 51.378823] R13: 0023 R14: 88000f1661b8 R15: 88001549c060 [ 51.378910] FS: 7fee7e146840() GS:880017c0() knlGS: [ 51.379038] CS: 0010 DS: ES: CR0: 80050033 [ 51.379118] CR2: 0023 CR3: 175f4000 CR4: 001407f0 [ 51.379238] Stack: [ 51.379299] 810fb39b 0004 88000f1633e8 [ 51.379523] 88001549c000 88001549c060 8800158f7df8 810fdb36 [ 51.379827] 08e6 00040023 88000f166180 88000f1637c0 [ 51.380015] Call Trace: [ 51.380085] [810fb39b] ? audit_compare_dname_path+0x2b/0xa0 [ 51.380170] [810fdb36] __audit_inode_child+0xb6/0x330 [ 51.380254] [811ca060] vfs_mknod+0x110/0x160 [ 51.380350] [816bf475] unix_bind+0x2a5/0x360 [ 51.380449] [810ff142] ? __audit_sockaddr+0x42/0x80 [ 51.380538] [8160d4c0] SYSC_bind+0xe0/0x120 [ 51.380617] [8160e4de] SyS_bind+0xe/0x10 [ 51.380777] [817330bd] system_call_fastpath+0x1a/0x1f [ 51.380943] Code: 89 f8 48 89 e5 f6 82 40 c7 84 81 20 74 15 0f 1f 44 00 00 48 83 c0 01 0f b6 10 f6 82 40 c7 84 81 20 75 f0 5d c3 66 0f 1f 44 00 00 80 3f 00 55 48 89 e5 74 15 48 89 f8 0f 1f 40 00 48 83 c0 01 80 [ 51.382310] RIP [8136cb80] strlen+0x0/0x30 [ 51.382432] RSP 8800158f7d60 [ 51.382518] CR2: 0023 [ 51.382654] ---[ end trace b5e4e75ed1093ea0 ]--- To Replicate: apt-get install auditd echo -w /etc/test /etc/audit/audit.rules /etc/init.d/auditd restart apt-get install linux-headers-3.13.0-51 linux-headers-3.13.0-51-generic linux-image-3.13.0-51-generic reboot attempt to login or ssh into the host - you'll get a similar stacktrace. ** Affects: linux-lts-trusty (Ubuntu) Importance: Undecided Status: New ** Package changed: linux (Ubuntu) = linux-lts-trusty (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1450643 Title: Installing linux-image-3.13.0-51-generic with auditd causes unrecoverable system state Status in linux-lts-trusty package in Ubuntu: New Bug description: When installing linux kernel 3.13.0-51 onto a system that has auditd installed with at least one rule enabled, the following stacktrace occurs when various processes occur (pretty much anything by root). [ 41.813593] type=1305 audit(1430428709.300:28): audit_pid=0 old=880 auid=4294967295 ses=4294967295 res=1 [ 41.830177] type=1305 audit(1430428709.316:29): audit_enabled=1 old=1 auid=900 ses=1 res=1 [ 41.830184] type=1305 audit(1430428709.316:30): audit_pid=1238 old=0 auid=900 ses=1 res=1 [ 51.377064] BUG: unable to handle kernel NULL pointer dereference at 0023 [ 51.377243] IP: [8136cb80] strlen+0x0/0x30 [ 51.377312] PGD 0 [ 51.377358] Oops: [#1] SMP [ 51.377428] Modules linked in: vmw_vsock_vmci_transport vsock dm_crypt ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
