[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
You can potentially use numactl to launch the process and set a policy of interleaving allocations between NUMA nodes to avoid these 1 sided allocations. Tends to happen with servers that make big allocations from a single thread during startup, as commonly seen on mysqld servers and the innodb_buffer_pool for example. numactl --interleave all /path/to/server/process --argument-1 #etc Reference: https://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
In our production environment of ~1800 nodes we've seen oom-kill events that looked similar to this bug's pattern - oom-kills killing large server processes while resident memory was far lower than available physical memory. We were affected by the original bug and saw that issue readily addressed in newer kernel versions, as mentioned in the earlier comments in this ticket. However, we still kept seeing oom-kill events, albeit in far lower numbers over time, that were happening on kernel-upgraded systems. These were a mystery for awhile, largely due to their infrequent occurrence. After a lot of research we think we've pinned it down to a subset of our multi-socket servers that have >1 NUMA memory pools. After implementing some scripts to track NUMA stats we've observed that one of the two NUMA pools is being fully utilized while the other has large amounts of memory to spare (often 90-95%) Either our server app, the JVM its running on, or the kernel itself isn't handling the NUMA memory pooling well and we're ending up exhausting an entire NUMA pool. Work is ongoing to see the causality chain that's leading to this. We don't yet have confirmation about whether its something our app (or its libraries) is doing, if we just need to make the JVM NUMA-aware with args, or if there's kernel tuning to be done. But I did want to mention it here as a warning to folks running on multi-NUMA-pool multi-socket systems seeing similar behavior. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Description changed: - I recently replaced some Xenial servers, and started experiencing "Out - of memory" problems with the default kernel. + After a fix for LP#1647400, a bug that caused freezes under some + workloads, some users noticed regular OOMs. Those regular OOMs were + reported under this bug, and fixed after some releases. + + Some of the affected kernels are documented below. In order to check + your particular kernel, read its changelog and lookup for 1655842 and + 1647400. If it has the fix for 1647400, but not the fix for 1655842, + then it's affected. + + It's still possible that you notice regressions compared to kernels that + didn't have the fixes for any of the bugs. However, reverting all fixes + would cause the freeze bug to come back. So, it's not a possible + solution moving forward. + + If you see any regressions, in the form of OOMs, mainly, please report a + new bug. Different workloads may require different solutions, and it's + possible that further fixes are needed, be them upstream or not. The + best way to get such fixes applied is reporting that under a new bug, + one that can be verified, so being able to reproduce the bug makes it + possible to verify the fixes really fix the identified bug. + + Kernels affected: + + linux 4.4.0-58, 4.4.0-59, 4.4.0-60, 4.4.0-61, 4.4.0-62. + linux-raspi2 4.4.0-1039 to 4.4.0-1042 and 4.4.0-1044 to 4.4.0-1071 + + + Particular kernels NOT affected by THIS bug: + + linux-aws + + To reiterate, if you find an OOM with an affected kernel, please upgrade. + If you find an OOM with a non-affected kernel, please report a new bug. We want to investigate it and fix it. + + + === + I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Side Question... Is there something I can specifically look for on a Ubuntu install that would indicate if that kernel has the fix or not. I assume the "Fix or Not" indicates are manually triggered and the fix could be out for AWS but not indicated. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We believe that we are experiencing this issue on kernel 4.4.0-1030-aws as well. We recently move from 14 LTS to 16LTS and are now experience oom kills. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Not sure if it's the same issue, but we had an unexpected OOM with Ubuntu 16.04.3 LTS, 4.4.0-91. Oct 31 23:52:25 db3 kernel: [6569272.882023] psql invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 ... Oct 31 23:52:25 db3 kernel: [6569272.882154] Mem-Info: Oct 31 23:52:25 db3 kernel: [6569272.882165] active_anon:38011018 inactive_anon:1422084 isolated_anon:0 Oct 31 23:52:25 db3 kernel: [6569272.882165] active_file:11699125 inactive_file:11727535 isolated_file:0 Oct 31 23:52:25 db3 kernel: [6569272.882165] unevictable:0 dirty:88019 writeback:2902991 unstable:23308 Oct 31 23:52:25 db3 kernel: [6569272.882165] slab_reclaimable:1455159 slab_unreclaimable:533985 Oct 31 23:52:25 db3 kernel: [6569272.882165] mapped:38499394 shmem:38495946 pagetables:33687177 bounce:0 Oct 31 23:52:25 db3 kernel: [6569272.882165] free:212612 free_pcp:0 free_cma:0 Oct 31 23:52:25 db3 kernel: [6569272.882172] Node 0 DMA free:13256kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Oct 31 23:52:25 db3 kernel: [6569272.882182] lowmem_reserve[]: 0 1882 193368 193368 193368 Oct 31 23:52:25 db3 kernel: [6569272.882188] Node 0 DMA32 free:768204kB min:316kB low:392kB high:472kB active_anon:8kB inactive_anon:32kB active_file:20kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2045556kB managed:1964868kB mlocked:0kB dirty:0kB writeback:44kB mapped:16kB shmem:12kB slab_reclaimable:729192kB slab_unreclaimable:35928kB kernel_stack:1920kB pagetables:415552kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Oct 31 23:52:25 db3 kernel: [6569272.882196] lowmem_reserve[]: 0 0 191486 191486 191486 Oct 31 23:52:25 db3 kernel: [6569272.882201] Node 0 Normal free:34260kB min:32432kB low:40540kB high:48648kB active_anon:58162056kB inactive_anon:2546400kB active_file:18254204kB inactive_file:18282192kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:199229440kB managed:196081724kB mlocked:0kB dirty:152124kB writeback:4685924kB mapped:58223800kB shmem:58229824kB slab_reclaimable:2362116kB slab_unreclaimable:1123984kB kernel_stack:11056kB pagetables:94580096kB unstable:22108kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Oct 31 23:52:25 db3 kernel: [6569272.882210] lowmem_reserve[]: 0 0 0 0 0 Oct 31 23:52:25 db3 kernel: [6569272.882215] Node 1 Normal free:34728kB min:32780kB low:40972kB high:49168kB active_anon:93882008kB inactive_anon:3141904kB active_file:28542276kB inactive_file:28627900kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:201326592kB managed:198178644kB mlocked:0kB dirty:199952kB writeback:6925996kB mapped:95773760kB shmem:95753948kB slab_reclaimable:2729328kB slab_unreclaimable:976028kB kernel_stack:6608kB pagetables:39753060kB unstable:71124kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Oct 31 23:52:25 db3 kernel: [6569272.882226] lowmem_reserve[]: 0 0 0 0 0 Oct 31 23:52:25 db3 kernel: [6569272.882230] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 13256kB Oct 31 23:52:25 db3 kernel: [6569272.882248] Node 0 DMA32: 121*4kB (UME) 95*8kB (UME) 5337*16kB (UME) 4229*32kB (UME) 2523*64kB (UME) 624*128kB (UME) 237*256kB (UME) 83*512kB (UM) 51*1024kB (UM) 73*2048kB (UM) 0*4096kB = 768204kB Oct 31 23:52:25 db3 kernel: [6569272.882268] Node 0 Normal: 8587*4kB (UM) 8*8kB (MH) 15*16kB (H) 8*32kB (H) 3*64kB (H) 1*128kB (H) 2*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 36252kB Oct 31 23:52:25 db3 kernel: [6569272.882284] Node 1 Normal: 9063*4kB (UM) 0*8kB 7*16kB (H) 7*32kB (H) 5*64kB (H) 2*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 37676kB Oct 31 23:52:25 db3 kernel: [6569272.882303] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Oct 31 23:52:25 db3 kernel: [6569272.882306] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Oct 31 23:52:25 db3 kernel: [6569272.882308] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Oct 31 23:52:25 db3 kernel: [6569272.882311] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Oct 31 23:52:25 db3 kernel: [6569272.882313] 61926313 total pagecache pages Oct 31 23:52:25 db3 kernel: [6569272.882315] 3557 pages in swap cache Oct 31 23:52:25 db3 kernel: [6569272.882318] Swap cache stats: add
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-aws (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-aws (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Changed in: linux-raspi2 (Ubuntu) Status: Confirmed => Fix Committed ** Changed in: linux-raspi2 (Ubuntu Xenial) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Attached kern.log ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4944721/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We have found this issue on 4.4.0-92 too. But only when the systems were put under stress. Reverting back to 4.4.0-57 resolved it. ** Attachment added: "dmesg.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4944720/+files/dmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
With the sysctl settings provided by Pete (vm.min_free_kbytes = 100 and vm.zone_reclaim_mode = 1), we've been running the linux-aws 4.4.0-1028.37 kernel successfully without an OOM killer invocation for about four days now. Previously we would have seen three or more occurrences of this per day, so it's a positive indication. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Attaching dmesg output ** Attachment added: "dmesg-kernel4.4.0-89.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4934014/+files/dmesg-kernel4.4.0-89.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We have another case of OOM at one of the host that we upgraded kernel to 4.4.0-89 a week ago. kern.log attached. ** Attachment added: "Kern log attached." https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4934013/+files/kern-log-kernel4.4.0-89.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
> kthreadd invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 Yea - that 2nd order allocation failure is the exact same issue I was able to see (same GFP mask also) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Apologies - the file was inadvertently split by logrotate. I have concatenated the entire contents of kern.log and kern.log.1 into the attached file; these are the only kern.log files in /var/log on the system. I do have to redact the hostname in question, but it is a simple substitution of 'localhost' for the FQDN of the system. ** Attachment added: "full kern.log contents" https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1655842/+attachment/4933724/+files/full_kern_20170816.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I have seemingly solved this issue with linux-aws version 4.4.0-1016-aws at the very least. The specific issue I was seeing was 2nd order allocations failing when OOMKiller triggered. At the time I was thinking the issue was due to XFS and memory fragmentation with lots and lots of memory mapped files in Elasticsearch/Lucene. When we moved to EXT4 the rate of oomkiller firing dropped, but did not stop. We made the following 2 changes to sysctls which have effectively stopped higher order memory allocaitons from failing and oomkiller firing. Also these settings were used on i3.2xlarge hosts that have 60G of ram - your milage may vary. Also we do not run swap on our servers, so likely adding swap could have helped, but not an option for us. vm.min_free_kbytes = 100 # We set this to leave about 1G of ram available for the kernel in the hope that even if the memory was heavily fragmented there might still be enough memory for linux to grab a higher order memory allocation fast enough before oomkiller does things. vm.zone_reclaim_mode = 1 # our hope here was to get the kernel to get more aggressive in reclaiming memory -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Please, do not cut the logs. Without the "invoked oom-killer" line, for example, it's hard to see the gfp flags and allocation order that failed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We are also experiencing this issue running linux-aws 4.4.0-1028.37, which tracks Ubuntu kernel 4.4.0-89.112. Our use case is very similar to comment #86 (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/comments/86). In our case ElasticSearch 2.4.5 is running under Java 1.8.0_131 with a ~29GB heap; we downsized from 31GB as a troubleshooting effort with no change to the frequency of OOM. The issue also occurs regardless of vm.overcommit_memory being set to 0, 1 or 2. The relevant data from kern.log (with redacted hostname) is attached; I'm happy to provide additional logs or test different kernels, but since our use case is i3-class instances in AWS, we need the nvme enhancements and enhanced network I/O provided by the linux-aws package. ** Attachment added: "kern.log from affected system" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4933628/+files/kern_20170816.log ** Also affects: linux-aws (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi I am experiencing at one of our compute node hypervisor. kernel version we are using is 4.4.0-83, but seems to be having the issue reported in this report. [Mon Aug 7 00:19:42 2017] nova-compute invoked oom-killer: gfp_mask=0x2c200ca, order=0, oom_score_adj=0 [Mon Aug 7 00:19:42 2017] nova-compute cpuset=/ mems_allowed=0-1 [Mon Aug 7 00:19:42 2017] CPU: 7 PID: 2164484 Comm: nova-compute Tainted: G OE 4.4.0-83-generic #106-Ubuntu [Mon Aug 7 00:19:42 2017] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.4.3 01/17/2017 [Mon Aug 7 00:19:42 2017] 0286 d6004dce 88014e753a50 813f9513 [Mon Aug 7 00:19:42 2017] 88014e753c08 883fecf88e00 88014e753ac0 8120b53e [Mon Aug 7 00:19:42 2017] 0015 881fe883b740 883fe94f7000 [Mon Aug 7 00:19:42 2017] Call Trace: [Mon Aug 7 00:19:42 2017] [] dump_stack+0x63/0x90 [Mon Aug 7 00:19:42 2017] [] ? apparmor_capable+0xc4/0x1b0 [Mon Aug 7 00:19:42 2017] [] oom_kill_process+0x202/0x3c0 [Mon Aug 7 00:19:42 2017] [] out_of_memory+0x219/0x460 [Mon Aug 7 00:19:42 2017] [] __alloc_pages_slowpath.constprop.88+0x938/0xad0 [Mon Aug 7 00:19:42 2017] [] __alloc_pages_nodemask+0x286/0x2a0 [Mon Aug 7 00:19:42 2017] [] alloc_pages_vma+0xad/0x250 [Mon Aug 7 00:19:42 2017] [] do_huge_pmd_wp_page+0x153/0xb70 [Mon Aug 7 00:19:42 2017] [] handle_mm_fault+0x90f/0x1820 [Mon Aug 7 00:19:42 2017] [] ? do_page_fault+0x22/0x30 [Mon Aug 7 00:19:42 2017] [] ? page_fault+0x28/0x30 [Mon Aug 7 00:19:42 2017] [] __do_page_fault+0x197/0x400 [Mon Aug 7 00:19:42 2017] [] do_page_fault+0x22/0x30 [Mon Aug 7 00:19:42 2017] [] page_fault+0x28/0x30 [Mon Aug 7 00:19:42 2017] Mem-Info: [Mon Aug 7 00:19:42 2017] active_anon:61350709 inactive_anon:2118817 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:32 unevictable:915 dirty:0 writeback:8 unstable:0 slab_reclaimable:14082 slab_unreclaimable:64456 mapped:3492 shmem:329012 pagetables:142167 bounce:0 free:260204 free_pcp:4111 free_cma:0 [Tue Aug 8 05:50:08 2017] apt-check invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [Tue Aug 8 05:50:08 2017] apt-check cpuset=/ mems_allowed=0-1 [Tue Aug 8 05:50:08 2017] CPU: 11 PID: 2538289 Comm: apt-check Tainted: G OE 4.4.0-83-generic #106-Ubuntu [Tue Aug 8 05:50:08 2017] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.4.3 01/17/2017 [Tue Aug 8 05:50:08 2017] 0286 5e467cc9 8820b44a39f8 813f9513 [Tue Aug 8 05:50:08 2017] 8820b44a3bb0 881fec15b800 8820b44a3a68 8120b53e [Tue Aug 8 05:50:08 2017] 0015 81e42ac0 883fe996f980 ff04 [Tue Aug 8 05:50:08 2017] Call Trace: [Tue Aug 8 05:50:08 2017] [] dump_stack+0x63/0x90 [Tue Aug 8 05:50:08 2017] [] ? cap_capable+0xd/0x70 [Tue Aug 8 05:50:08 2017] [] oom_kill_process+0x202/0x3c0 [Tue Aug 8 05:50:08 2017] [] out_of_memory+0x219/0x460 [Tue Aug 8 05:50:08 2017] [] __alloc_pages_slowpath.constprop.88+0x938/0xad0 [Tue Aug 8 05:50:08 2017] [] __alloc_pages_nodemask+0x286/0x2a0 [Tue Aug 8 05:50:08 2017] [] alloc_pages_current+0x8c/0x110 [Tue Aug 8 05:50:08 2017] [] __page_cache_alloc+0xab/0xc0 [Tue Aug 8 05:50:08 2017] [] filemap_fault+0x14a/0x3f0 [Tue Aug 8 05:50:08 2017] [] ext4_filemap_fault+0x36/0x50 [Tue Aug 8 05:50:08 2017] [] __do_fault+0x50/0xe0 [Tue Aug 8 05:50:08 2017] [] handle_mm_fault+0xfa2/0x1820 [Tue Aug 8 05:50:08 2017] [] __do_page_fault+0x197/0x400 [Tue Aug 8 05:50:08 2017] [] do_page_fault+0x22/0x30 [Tue Aug 8 05:50:08 2017] [] page_fault+0x28/0x30 [Tue Aug 8 05:50:08 2017] Mem-Info: [Tue Aug 8 05:50:08 2017] active_anon:61377850 inactive_anon:2049156 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:915 dirty:0 writeback:0 unstable:0 slab_reclaimable:15329 slab_unreclaimable:101408 mapped:3655 shmem:338468 pagetables:141874 bounce:0 free:260450 free_pcp:2714 free_cma:0 Linux rcgpudc1rh31-02 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Mine's also still stable (no OOMs), after running the patched kernel for 9 days, on a Raspberry pi 2 Model B v1.1. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
4.4.0-1062-raspi2 is looking good - I've had it running for a week without oom-killer being invoked. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Update: No sign of Out-of-memory errors or kills, after 3 days of testing the 4.4.0-1062-raspi2 kernel. I'll report back again next week. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Sure. I undid the workaround, installed and booted the kernel and will test it for a few days. I'll keep you posted on results. Thanks Paolo! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Chris, can you test if this kernel solves your problem? http://people.canonical.com/~ppisati/lp1655842/linux- image-4.4.0-1062-raspi2_4.4.0-1062.70~lp1655842_armhf.deb -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Changed in: linux-raspi2 (Ubuntu) Assignee: (unassigned) => Paolo Pisati (p-pisati) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
The bug is still confirmed and not fixed for linux-raspi2 (Ubuntu), 5 months after getting fixed for the main Ubuntu. Shouldn't this have some priority? Even apt upgrade breaks if I don't use the clear cache workaround. I can live with it (cron job to clear cache) but this is not great for LTS. Currently affected: Ubuntu 16.04.2 LTS, 4.4.0-1059-raspi2 #67-Ubuntu -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@nicholas-hatch - what file system are your disks formatted as? I was able to stop the OOM's on my ES hosts by moving from XFS to EXT4. My belief is that there was a memory fragmentation issue with ES and many small files on XFS formatted volumes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We're still having issues with higher-order allocations failing and triggering an OOM kill for unexplainable reasons. (on 4.4.0-78-generic). I've attached the relevant OOM killer logs. It may be relevant to note that the server these logs are from is an Elasticsearch instance with a large (~32GB) mlock'ed heap. ** Attachment added: "kthreadd triggered order=2 allocation failure" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4897378/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Also observed with 4.4.0-1054-raspi2. I'm now back on 4.4.0-1038-raspi2. I think that one was ok. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Changed in: linux-raspi2 (Ubuntu) Status: New => Confirmed ** Changed in: linux-raspi2 (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I'm seeing oom-killer being invoked despite having 2GB free swap when using the kernel from linux-image-4.4.0-1055-raspi2 version 4.4.0-1055.62. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
linux-raspi2 version 4.4.0.1055.56 that is. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is still an issue in the current linux-raspi2 version. Where those changes ported to that kernel? ** Also affects: linux-raspi2 (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I'd like to emphasise that the OOM problem only happens with KDE. I have several DE installed including Unity, GNOME3, Cinnamon. But none of them caused a OOM, at least I never noticed. But in KDE, most of the time when chrome is opened, it triggers OOM. dmesg tells that, sometimes kwin_x11 invoked the OOM or plasmashell. Most of the time plasmashell is crashed and the opened tab in chrome is killed. However, chrome application will be there. I need to start plasmashell by pressing Alt-F2 bringing the run command dialog and type plasmashell there. Last night, Even firefox gave an OOM. I'm attaching a dmesg log hoping that will be helpful. ** Attachment added: "dmesg log in KDE with 4.4.0-21 kernel" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4859987/+files/shitty-dmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I also haven't personally encountered any further OOM issues on my home desktop (used daily) with 4.4.0.63. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Yes, 4.4.0.63 solved our OOM issues (and we had plenty after 4.4.0.59). Ours were all headless server (bare metal and VMs) related though - no desktop usage. But I never experienced this issue with my home laptop either, but that had lots of RAM and was only lightly used during that period - not really a good data point. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Anybody having successfully tested 4.4.0.63 for OOM-kill issue... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I have 4.4.0-59 till 4.4.0-71 and 4.8.0-41 till 4.8.0-46 installed on my system and all are affected by this bug. Firefox, Chrome and Netbeans regularly get killed without a warning and for no reason (since I have something like 10GB+ RAM and all 16GB Swap free at the time the process gets killed). Even KDE has been killed a couple of times while the system still had over 6GB RAM and 16GB Swap free. Yesterday, after the umpteenth time Netbeans was killed while I was in the middle of doing something, I finally decided to do something about this problem and installed Kernel 4.10.9-041009 from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10.9/. Sadly, that doesn't seem to resolve the problem either and Oom Killer is still overeager to kill user processes (Firefox and Netbeans have both been killed multiple times). At least KDE hasn't been killed so far. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I was using 4.4.0-21 as reported by `uname -r` which is default in Kubuntu 16.04. The same bug appears on mainline kernel 4.10 too! Now, I'm in confusion. Which kernel should I upgrade to? Also I experience this only in KDE session with yandex or chrome browser opened -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is plaguing Horizon project Pi2 and Pi3 devices running Xenial 16.04.2 w/ kernel 4.4.0-1050-raspi2. From a pi2: root@horizon-a17d2187:~# uname -a Linux horizon-a17d2187 4.4.0-1050-raspi2 #57-Ubuntu SMP Wed Mar 22 12:52:22 UTC 2017 armv7l armv7l armv7l GNU/Linux root@horizon-a17d2187:~# free totalusedfree shared buff/cache available Mem: 942128 149548 35456 494084 757124 239716 Swap: 0 0 0 Under these circumstances, the kernel's oom-killer will kill Wifi processes (rtl_rpcd), systemd-udevd, our Ethereum client (geth), and other critical processes in attempt to stay afloat rather than using reclaimable RAM. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I am seeing this issue as well, on Arch kernel v 4.10.4-1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
We have been seeing this issue recently as well. We are running 4.4.0-66-generic #87-Ubuntu - I can attempt to downgrade to 4.4.0-57 but its a large cluster with a lot of data so it may take some time. Attached a kern.log from this most recent oom. ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4841016/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi Cascardo, there is no related dmesg output after the incident (just some lines from apt-daily.timer). I filed a bug for the possible swap corruption issue here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1669707 Thanks so far. Julian -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Julian, your logs indicate some possible swap corruption, would you mind opening a new bug and sending it using apport-bug? Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi, Julian. Do you have the output of dmesg after the incident before a reboot? Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Attached kern.log. Let me know in case you need more logs. ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4829766/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi, Anton and Julian. Can you attach complete logs for investigation? Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Same here on Linux 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux. Out of memory: Kill process 11067 (java) score 28 or sacrifice child Killed process 11067 (java) total-vm:3569724kB, anon-rss:211720kB, file-rss:20208kB systemd-journald[247]: /dev/kmsg buffer overrun, some messages lost. swap_free: Bad swap file entry 2000 BUG: Bad page map in process java pte:0020 pmd:dac28067 addr:7fc34ce69000 vm_flags:0871 anon_vma: (null) mapping: (null) index:7fc34ce69 file: (null) fault: (null) mmap: (null) readpage: (null) CPU: 1 PID: 11108 Comm: java Tainted: GB D 4.4.0-64-generic #85-Ubuntu Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 0286 f2f56de1 8800d8ceba58 813f8083 7fc34ce69000 8800dac023e8 8800d8cebaa8 811be06f 8800d8ceba80 811d518c 2000 0020 Call Trace: [] dump_stack+0x63/0x90 [] print_bad_pte+0x1df/0x2a0 [] ? swap_info_get+0x7c/0xd0 [] unmap_page_range+0x468/0x7a0 [] unmap_single_vma+0x7d/0xe0 [] unmap_vmas+0x51/0xa0 [] exit_mmap+0xa7/0x170 [] mmput+0x57/0x130 [] do_exit+0x27a/0xb00 [] ? __unqueue_futex+0x2c/0x60 [] ? futex_wait+0x16e/0x280 [] do_group_exit+0x43/0xb0 [] get_signal+0x292/0x600 [] do_signal+0x37/0x6f0 [] ? mntput+0x24/0x40 [] ? __fput+0x190/0x220 [] exit_to_usermode_loop+0x8c/0xd0 [] syscall_return_slowpath+0x4e/0x60 [] int_ret_from_sys_call+0x25/0x8f BUG: Bad rss-counter state mm:8800d8bc8800 idx:2 val:-1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I still see OOM killer invocation on 4.4.0-64.85: Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857840] java invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857846] java cpuset=/ mems_allowed=0-1 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857855] CPU: 27 PID: 47820 Comm: java Tainted: GW 4.4.0-64-generic #85-Ubuntu Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857857] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/12/2016 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857860] 0286 ee496386 882358f13b10 813f8083 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857867] 882358f13cc8 883c70a8f000 882358f13b80 8120b0fe Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857871] 81cd63bf 81e677e0 0206 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857875] Call Trace: Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857885] [] dump_stack+0x63/0x90 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857893] [] dump_header+0x5a/0x1c5 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857899] [] oom_kill_process+0x202/0x3c0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857902] [] out_of_memory+0x219/0x460 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857907] [] __alloc_pages_slowpath.constprop.88+0x938/0xad0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857911] [] __alloc_pages_nodemask+0x286/0x2a0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857915] [] alloc_kmem_pages_node+0x4b/0xc0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857921] [] copy_process+0x1be/0x1b70 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857930] [] ? handle_mm_fault+0xcbd/0x1820 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857935] [] ? call_rwsem_down_read_failed+0x14/0x30 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857938] [] _do_fork+0x80/0x360 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857942] [] SyS_clone+0x19/0x20 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857948] [] entry_SYSCALL_64_fastpath+0x16/0x71 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857969] Mem-Info: Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] active_anon:32518350 inactive_anon:2099 isolated_anon:0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] active_file:45384948 inactive_file:45384381 isolated_file:64 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] unevictable:914 dirty:104 writeback:0 unstable:0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] slab_reclaimable:1282591 slab_unreclaimable:39566 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] mapped:10291496 shmem:2227 pagetables:732932 bounce:0 Feb 27 15:43:09 ip-10-0-9-47 kernel: [61385.857981] free:272957 free_pcp:1153 free_cma:0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I can confirm 4.4.0-64.85 fixes our OOM issues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Could somebody please confirm that the issue has been fixed with the kernel 4.4.0-64.85? We had massive problems with OOM killed qemu instances on our virtualization hosts and would not like to have this introduced again. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I'm pretty sure this bug was introduced in 4.4.0-58, even though I reported it against 4.4.0-59. Still, +1 for a raising a separate bug against 4.8. Thanks to Cascardo for fixing this one. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
The OOM fixes were introduced in 4.4.0-58 according to the changelog, but sure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi, Mr. Iain Buclaw. Memory usage reports could be related to something else. This bug was introduced at 4.4.0-59, you mention 4.4.0-58. We could certainly investigate the issue you see, just not on this bug. Much more data is necessary, but, please, don't attach new data to this bug. Your report relates to a different kernel, which has a very different memory management code. I could ask you to test 4.4.0-63, but it still is possible that you find problems there, because they are unrelated to this bug. Thank you. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is the reported /proc/meminfo Buffers usage for 4 different kernel versions. We got the same OOM call traces on both 4.4.0-58 and 4.8.0-34, I highly doubt that to be a coincidence. ** Attachment added: "meminfo-buffers.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4822727/+files/meminfo-buffers.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This bug is not about memory use. It's about the Linux kernel triggering the OOM killer when higher-order (order 2) allocations are requested and progress cannot be made. This affected Linux 4.7 on mainline and was fixed on Linux 4.7 stable and Linux 4.8. When some fixes were backported to 4.4.0-59 (4.4.0-58 was not affected), this bugs was introduced to Xenial kernels and is now fixed on 4.4.0-63. Any behavior on 4.8 kernels must be investigated separately, because all fixes that were backported to 4.4.0-63 are already present in 4.8. Can you please open a new bug and attach all logs and details you can, so we can investigate your problem and provide a fix? Please, do not use this bug, because the fixes would be different anyway, and even thought the symptom may look like, we consider them different bugs. I appreciate you opening a new bug and providing this new report. Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Yes they are, I'm seeing the same exorbitant memory usage that we had on 4.4.0-58 as am currently getting on 4.8.0-36. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
4.8 kernels are not affected by this bug. If you have OOMs on 4.8 kernels, please file a new bug with all the revelant details and logs. Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Bug is prevalent on 4.8 kernels too. --- [251529.693133] CPU: 3 PID: 1547 Comm: icinga2 Not tainted 4.8.0-34-generic #36~16.04.1-Ubuntu [251529.693134] Hardware name: MSI MS-7823/B85M-G43 (MS-7823), BIOS V3.14B3 06/23/2014 [251529.693135] 0286 e9fa7ede 95c3f774bb38 ab62d7b3 [251529.693137] 95c3f774bcc8 95c3f23bd700 95c3f774bba0 ab42e9bb [251529.693138] 95c3f774bb40 [251529.693140] Call Trace: [251529.693145] [] dump_stack+0x63/0x90 [251529.693148] [] dump_header+0x5c/0x1dc [251529.693151] [] oom_kill_process+0x226/0x3f0 [251529.693153] [] out_of_memory+0x35a/0x3f0 [251529.693155] [] __alloc_pages_slowpath+0x9fb/0xa20 [251529.693157] [] __alloc_pages_nodemask+0x2ba/0x300 [251529.693160] [] copy_process.part.30+0x146/0x1b50 [251529.693162] [] ? sock_recvmsg+0x3d/0x50 [251529.693163] [] ? SYSC_recvfrom+0xda/0x150 [251529.693164] [] _do_fork+0xe7/0x3f0 [251529.693166] [] ? __sys_recvmsg+0x51/0x90 [251529.693168] [] SyS_clone+0x19/0x20 [251529.693170] [] do_syscall_64+0x5e/0xc0 [251529.693174] [] entry_SYSCALL64_slow_path+0x25/0x25 [251529.693174] Mem-Info: [251529.693177] active_anon:339565 inactive_anon:133615 isolated_anon:0 active_file:3938458 inactive_file:328087 isolated_file:0 unevictable:8 dirty:200 writeback:37 unstable:0 slab_reclaimable:3365424 slab_unreclaimable:16102 mapped:9114 shmem:1459 pagetables:2462 bounce:0 free:49449 free_pcp:32 free_cma:0 --- Had 5 servers knocked out over the weekend. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This bug was fixed in the package linux - 4.4.0-63.84
---
linux (4.4.0-63.84) xenial; urgency=low
[ Thadeu Lima de Souza Cascardo ]
* Release Tracking Bug
- LP: #1660704
* Backport Dirty COW patch to prevent wineserver freeze (LP: #1658270)
- SAUCE: mm: Respect FOLL_FORCE/FOLL_COW for thp
* Kdump through NMI SMP and single core not working on Ubuntu16.10
(LP: #1630924)
- x86/hyperv: Handle unknown NMIs on one CPU when unknown_nmi_panic
- SAUCE: hv: don't reset hv_context.tsc_page on crash
* [regression 4.8.0-14 -> 4.8.0-17] keyboard and touchscreen lost on Acer
Chromebook R11 (LP: #1630238)
- [Config] CONFIG_PINCTRL_CHERRYVIEW=y
* Call trace when testing fstat stressor on ppc64el with virtual keyboard and
mouse present (LP: #1652132)
- SAUCE: HID: usbhid: Quirk a AMI virtual mouse and keyboard with
ALWAYS_POLL
* VLAN SR-IOV regression for IXGBE driver (LP: #1658491)
- ixgbe: Force VLNCTRL.VFE to be set in all VMDq paths
* "Out of memory" errors after upgrade to 4.4.0-59 (LP: #1655842)
- mm, page_alloc: convert alloc_flags to unsigned
- mm, compaction: change COMPACT_ constants into enum
- mm, compaction: distinguish COMPACT_DEFERRED from COMPACT_SKIPPED
- mm, compaction: simplify __alloc_pages_direct_compact feedback interface
- mm, compaction: distinguish between full and partial COMPACT_COMPLETE
- mm, compaction: abstract compaction feedback to helpers
- mm, oom: protect !costly allocations some more
- mm: consider compaction feedback also for costly allocation
- mm, oom, compaction: prevent from should_compact_retry looping for ever
for
costly orders
- mm, oom: protect !costly allocations some more for !CONFIG_COMPACTION
- mm, oom: prevent premature OOM killer invocation for high order request
* Backport 3 patches to fix bugs with AIX clients using IBMVSCSI Target Driver
(LP: #1657194)
- SAUCE: ibmvscsis: Fix max transfer length
- SAUCE: ibmvscsis: fix sleeping in interrupt context
- SAUCE: ibmvscsis: Fix srp_transfer_data fail return code
* NVMe: adapter is missing after abnormal shutdown followed by quick reboot,
quirk needed (LP: #1656913)
- nvme: apply DELAY_BEFORE_CHK_RDY quirk at probe time too
* Ubuntu 16.10 KVM SRIOV: if enable sriov while ping flood is running ping
will stop working (LP: #1625318)
- PCI: Do any VF BAR updates before enabling the BARs
- PCI: Ignore BAR updates on virtual functions
- PCI: Update BARs using property bits appropriate for type
- PCI: Separate VF BAR updates from standard BAR updates
- PCI: Don't update VF BARs while VF memory space is enabled
- PCI: Remove pci_resource_bar() and pci_iov_resource_bar()
- PCI: Decouple IORESOURCE_ROM_ENABLE and PCI_ROM_ADDRESS_ENABLE
- PCI: Add comments about ROM BAR updating
* Linux rtc self test fails in a VM under xenial (LP: #1649718)
- kvm: x86: Convert ioapic->rtc_status.dest_map to a struct
- kvm: x86: Track irq vectors in ioapic->rtc_status.dest_map
- kvm: x86: Check dest_map->vector to match eoi signals for rtc
* Xenial update to v4.4.44 stable release (LP: #1658091)
- Input: xpad - use correct product id for x360w controllers
- Input: i8042 - add Pegatron touchpad to noloop table
- selftests: do not require bash to run netsocktests testcase
- selftests: do not require bash for the generated test
- mm: fix devm_memremap_pages crash, use mem_hotplug_{begin, done}
- ocfs2: fix crash caused by stale lvb with fsdlm plugin
- mm/hugetlb.c: fix reservation race when freeing surplus pages
- KVM: x86: fix emulation of "MOV SS, null selector"
- KVM: eventfd: fix NULL deref irqbypass consumer
- jump_labels: API for flushing deferred jump label updates
- KVM: x86: flush pending lapic jump label updates on module unload
- KVM: x86: add Align16 instruction flag
- KVM: x86: add asm_safe wrapper
- KVM: x86: emulate FXSAVE and FXRSTOR
- KVM: x86: Introduce segmented_write_std
- nl80211: fix sched scan netlink socket owner destruction
- USB: serial: kl5kusb105: fix line-state error handling
- USB: serial: ch341: fix initial modem-control state
- USB: serial: ch341: fix open error handling
- USB: serial: ch341: fix control-message error handling
- USB: serial: ch341: fix open and resume after B0
- Input: elants_i2c - avoid divide by 0 errors on bad touchscreen data
- i2c: print correct device invalid address
- i2c: fix kernel memory disclosure in dev interface
- xhci: fix deadlock at host remove by running watchdog correctly
- vme: Fix wrong pointer utilization in ca91cx42_slave_get
- mnt: Protect the mountpoint hashtable with mount_lock
- tty/serial: atmel_serial: BUG: stop DMA from transmitting in stop_tx
- sysrq: attach sysrq handler correctly for 32-bit kernel
- sysctl: Drop reference added by grab_header in
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Description changed: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 Jan 12 06:29 seq - crw-rw 1 root audio 116, 33 Jan 12 06:29 timer + total 0 + crw-rw 1 root audio 116, 1 Jan 12 06:29 seq + crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: - + ProcEnviron: - TERM=xterm-256color - PATH=(custom, no user) - XDG_RUNTIME_DIR= - LANG=en_US.UTF-8 - SHELL=/bin/bash + TERM=xterm-256color + PATH=(custom, no user) + XDG_RUNTIME_DIR= + LANG=en_US.UTF-8 + SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: - linux-restricted-modules-4.4.0-59-generic N/A - linux-backports-modules-4.4.0-59-generic N/A - linux-firmware1.157.6 + linux-restricted-modules-4.4.0-59-generic N/A + linux-backports-modules-4.4.0-59-generic N/A + linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I was curious if I could answer Sridhar's question as I had the same question. The introduction of the problem appears to be in Ubuntu's packaging of select upstream commits from 4.7 to address bug #1647400. >From the comments in that case, it appears 6b4e3181d7bd5ca5ab6f45929e4a5ffa7ab4ab7f should have also been selected. >From looking at the history I gather if I had been running stock 4.4 kernels I would not have been effected by the OOM issue. I'm basing this on tracking down 0a0337e0d1d134465778a16f5cbea95086e8e9e0 in the main line kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Did anybody also notice any pattern in "invoked oom-killer". I see a pattern like daily almost same time cron or scripts invoking oom- killer... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Thanks to those who reported/fixed this bug. Out of curiosity, was this a bug in the 4.4.0-59 kernel itself or in Ubuntu's packaging of the kernel, i.e. were other (non-Ubuntu) Linux users impacted ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
To throw some more light into the matter. Two machines were upgraded to 4.4.0-59 on 16th Jan, same load, but only one out of them is reporting oom-kill. Anybody experiencing same scenario... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Like Luk (#29), I upgraded to 4.8.0-34, but the problem disappeared for me. I ran system for two days without any OOM-kills. Before that, simply copying a big file (9Gb+) fired the system. My system has 16Gb RAM and runs 16.04.1. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
you should re-release 4.4.0-62 as linux-image-chaosmonkey-virtual -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Tested Ubuntu 4.4.0-63.84-generic 4.4.44 on a desktop system with a workload which previously led to Chrome processes being OOM-killed. Situation with 4.4.0-62-generic: between 8 and 54 processes OOM-killed per 24-hour period Situation with 4.4.0-63-generic: no OOM-kills during 46 hours of testing Look solved. No negative side-effects encountered. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Tags removed: verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
-63 works for me as well, thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I've been using -63 for a while now (even before it was in proposed, via an sbuild setup) on a machine that had OOM problems with -59, and I haven't noticed any issues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Firefox and Insync were killed pretty reliably for me, but other packages as well. I was getting half-a-dozen oom kills a day before I switched back to 57. My second workstation is fully updated and doesn't show any sign of this issue. >From m3 note Original message Sender: Nate Eldredge <1655...@bugs.launchpad.net> Time: Fri 2/10 19:21 To: wurlyfan <wurly...@gmail.com> Subject: Re: [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59 >On Fri, 10 Feb 2017, Travisgevans wrote: > >> However, it might help if anyone has an idea how the OOM kill bug might >> be reliably reproduced. “5 working days” isn't very long to reliably be >> sure the problem is solved otherwise; it took more than half that time >> upon upgrading to -59 for me to hit the bug by chance. > >I had a job (duplicity) that would oom every time under -59 and -62. With >-63 from proposed, it doesn't. > >-- >Nate Eldredge >n...@thatsmathematics.com > >-- >You received this bug notification because you are subscribed to the bug >report. >https://bugs.launchpad.net/bugs/1655842 > >Title: > "Out of memory" errors after upgrade to 4.4.0-59 > >Status in linux package in Ubuntu: > Fix Released >Status in linux source package in Xenial: > Fix Committed > >Bug description: > I recently replaced some Xenial servers, and started experiencing "Out > of memory" problems with the default kernel. > > We bake Amazon AMIs based on an official Ubuntu-provided image (ami- > e6b58e85, in ap-southeast-2, from https://cloud- > images.ubuntu.com/locator/ec2/). Previous versions of our AMI > included "4.4.0-57-generic", but the latest version picked up > "4.4.0-59-generic" as part of a "dist-upgrade". > > Instances booted using the new AMI have been using more memory, and > experiencing OOM issues - sometimes during boot, and sometimes a while > afterwards. An example from the system log is: > > [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' >at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. > [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan >2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds > [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or >sacrifice child > [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, >anon-rss:51184kB, file-rss:2164kB > [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or >sacrifice child > [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, >anon-rss:6676kB, file-rss:0kB > [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or >sacrifice child > [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, >anon-rss:23956kB, file-rss:1356kB > > I have a hunch that this may be related to the fix for > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, > introduced in linux (4.4.0-58.79). > > ProblemType: Bug > DistroRelease: Ubuntu 16.04 > Package: linux-image-4.4.0-59-generic 4.4.0-59.80 > ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 > Uname: Linux 4.4.0-59-generic x86_64 > AlsaDevices: > total 0 > crw-rw 1 root audio 116, 1 Jan 12 06:29 seq > crw-rw 1 root audio 116, 33 Jan 12 06:29 timer > AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' > ApportVersion: 2.20.1-0ubuntu2.4 > Architecture: amd64 > ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' > AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', >'/dev/snd/timer'] failed with exit code 1: > Date: Thu Jan 12 06:38:45 2017 > Ec2AMI: ami-0f93966c > Ec2AMIManifest: (unknown) > Ec2AvailabilityZone: ap-southeast-2a > Ec2InstanceType: t2.nano > Ec2Kernel: unavailable > Ec2Ramdisk: unavailable > IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' > Lsusb: Error: command ['lsusb'] failed with exit code 1: > MachineType: Xen HVM domU > PciMultimedia: > > ProcEnviron: > TERM=xterm-256color > PATH=(custom, no user) > XDG_RUNTIME_DIR= > LANG=en_US.UTF-8 > SHELL=/bin/bash > ProcFB: 0 cirrusdrmfb > ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic >root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 > RelatedPackageVersions: > linux-restricted-modules-4.4.0-59-generic N/A > linux-backports-modules-4.4.0-59-generic N/A > linux-firmware 1.157.6 > RfKill: Error: [Errno 2] No such file or directory: 'rfkill' > SourcePackage: linux > UpgradeStatus: No upgrade log pr
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I see this bug on Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-1042-raspi2 armv7l) as described here: https://superuser.com/questions/1176773 /ubuntu-on-rpi-starts-killing-processes-when-ram-is-filled-up-by-cache The workaround by Krzysztof Dryja (cih997) works for me as well but is very ugly and temporary. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
On Fri, 10 Feb 2017, Travisgevans wrote: > However, it might help if anyone has an idea how the OOM kill bug might > be reliably reproduced. “5 working days” isn't very long to reliably be > sure the problem is solved otherwise; it took more than half that time > upon upgrading to -59 for me to hit the bug by chance. I had a job (duplicity) that would oom every time under -59 and -62. With -63 from proposed, it doesn't. -- Nate Eldredge n...@thatsmathematics.com -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Don't forget that the earlier kernels are affected by Bug #1647400, which does something even worse (hang the system). I've verified that it affected my particular system before 4.4.0-59, and it may explain a couple of lockups I had previously experienced during normal operation when using previous kernels. -59 fixes the bug but introduces the permature OOM kill issue; if it weren't for the kernels currently in proposed (assuming they indeed fix this bug), I wouldn't really have a reliable kernel at all to use. With the 4.4.0-59 kernel, I got hit with two unexplained OOM kills, each occurring within about 3 days of uptime. I then tested the -62 kernel in proposed for just under 14 days and didn't see any OOM kills, and I've now been testing -63 for a couple of days and haven't any issues yet. However, it might help if anyone has an idea how the OOM kill bug might be reliably reproduced. “5 working days” isn't very long to reliably be sure the problem is solved otherwise; it took more than half that time upon upgrading to -59 for me to hit the bug by chance. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
To be fair, there have been multiple USN-reported kernel security patches fixed in post-57 kernels. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@nick-maynard: Why is such a bug unforgivable? You can just boot a previous kernel instead. If you're concerned about availability then don't reboot in the first place unless there's an important security patch. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Cascardo: Just to be clear, are you looking for verification from anyone in the world, or from specific kernel testers? (I'd like to help, but I'm only able to reproduce the issue in production, and the process of debugging this issue when we ran into it was already more restarts than is good for my service right now (we settled on downgrading for the moment).) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed- xenial'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
kulwinder singh: Either one, but nothing in between. -57 will reintroduce a few (unrelated) security bugs as well as the bug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400 whose fix caused this one, but is easier to enable and has been tested for longer. -63 should fix this bug, the older bug, and the intermediary security bugs, but requires you to enable the "proposed" repository, and hasn't been tested for quite as long. Anything in between has this bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Who is the saver 4.4.0-57-generic or 4.4.0-63-generic now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Testing... Enabled proposed (https://wiki.ubuntu.com/Testing/EnableProposed). Installed kernel packages: # apt-get install -s -t xenial-proposed 'linux-headers-4.4.0.63$' 'linux-headers-4.4.0.63-generic$' 'linux-image-4.4.0.63-generic$' 'linux-image-extra-4.4.0.63-generic$' Rebooted. # cat /proc/version_signature Ubuntu 4.4.0-63.84-generic 4.4.44 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
The fix is now available for testing in kernel version 4.4.0-63.84, if you enable proposed[1] $ apt-cache policy linux-image-4.4.0-63-generic linux-image-4.4.0-63-generic: Installed: (none) ==> Candidate: 4.4.0-63.84 Version table: 4.4.0-63.84 500 500 http://archive.ubuntu.com/ubuntu ==>xenial-proposed/main amd64 Packages $ apt-get changelog linux-image-4.4.0-63-generic | egrep "1655842" ==> * "Out of memory" errors after upgrade to 4.4.0-59 (LP: #1655842) [1] - https://wiki.ubuntu.com/Testing/EnableProposed - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
If this kernel is not going to hit -updates shortly (i.e. days), can something be done to pull or downgrade the broken kernel? At least revert linux-image-generic to depend back on linux- image-4.4.0-57-generic which doesn't have the issues and will stop more people from upgrading to a broken kernel. Having this sort of break in an LTS kernel is not inspiring at all. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
LTS Ubuntu with -updates shouldn't have this sort of issue - this is, frankly, unforgivable. We need a new kernel urgently in -updates, and I'd expect serious discussions within the kernel team to understand what has caused this issue and avoid it reoccurring. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Just by the way - 4.4.0-62-generic has the exact same problem. Even when uninstalling 4.4.0-59-generic, my system at some point auto-updated to 4.4.0-62-generic . Only 4.4.0-57-generic is safe for now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Hi, Luk. linux-image-4.8.0-34-generic should not be affected by this. If you see unexpected OOM problems, please open a new bug report and attach the kernel logs, please. Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This bug also appears to affect linux-image-4.8.0-34-generic in 16.04.1 Xenial. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
When this new kernel will be released? This bug is killing our MySQL Servers. Booting old kernels is only a bad workaround. I think a lot of people with busy servers will have a problem. This is 2.nd time we were hit by a big bug within short time. In oct 2016 our nameservers got problems because of bug 1634892 Is LTS-Ubuntu still right system for servers? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I've been struggling with this bug for nearly a week and only now found this issue. Thanks for fixing it! For the sake of others finding it, here's the stack trace part of the oom-killer log, which contains some terms I searched for a while ago that aren't mentioned here yet. docker invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=-1000 docker cpuset=/ mems_allowed=0 CPU: 11 PID: 4472 Comm: docker Tainted: GW 4.4.0-62-generic #83-Ubuntu Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 0286 57f64c94 880dfb5efaf0 813f7c63 880dfb5efcc8 880fbfda 880dfb5efb60 8120ad4e 81cd2d7f 81e67760 0206 Call Trace: [] dump_stack+0x63/0x90 [] dump_header+0x5a/0x1c5 [] oom_kill_process+0x202/0x3c0 [] out_of_memory+0x219/0x460 [] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70 [] __alloc_pages_nodemask+0x286/0x2a0 [] alloc_kmem_pages_node+0x4b/0xc0 [] copy_process+0x1be/0x1b70 [] ? apparmor_file_alloc_security+0x5c/0x220 [] ? kmem_cache_alloc+0x1ca/0x1f0 [] ? security_file_alloc+0x33/0x50 [] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [] _do_fork+0x80/0x360 [] SyS_clone+0x19/0x20 [] entry_SYSCALL_64_fastpath+0x16/0x71 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
As a note: I believe this also affects the armhf kernel 4.4.0-1040-raspi2 for the Raspberry Pi. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@nate Thank you! You just saved me a lot of hassle as I was about to unpin the 4.4.0-57 kernel and update a bunch of machines on the assumption the fix was in that version. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is fixed in 4.4.0-63.84, which will be available in proposed soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
I could not reboot my machine and the ugly workaround for this issue was to login as root and clear system caches: echo 3 > /proc/sys/vm/drop_caches This made my machine stable again, at least for the time I needed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Just as a note for newcomers reading this, I can confirm the bug is NOT fixed in the officially released 4.4.0-62.83. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is a severe bug. It should be treated a high-priority bugfix that cannot wait 3 weeks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@Gaudenz Steinlin (gaudenz-debian), It will takes 3 weeks to land in -updates pocket, but you can expect to have a call for testing a proposed package by EOW. - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@slashd It sounds really strange to me that I should wait til 20-Feb for a fix for this bug while this is clearly a regression introduced with the latest kernel upgrade. Is there no way to speed things up to fix this regression. Currently we had to downgrade all our xenial systems to linux- image-4.4.0-57-generic to avoid this bug. Gaudenz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
