[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Instant-update: So indeed I had a broken memory module. I had installed 2x1GB, but only 1 showed. (the 860 is due to the video controller using 200-odd MB system RAM) Now back up and running with 4GB . Will report back if I encounter the issue again (I haven't put the crontab fix in place so

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

FWIW this affects me too. I installed pretty much exactly the same installation (Ubuntu + Kodi + Steam) on two Mac minis, two days ago. You can compare the devices if you wish - MacMini 4,1 and 5,1 respectively. 4,1 works fine, 5,1 does not. https://en.wikipedia.org/wiki/Mac_Mini#Specifications_3

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

The bisect indicated the following commit as the first bad commit: commit 519192aaae38e24d6b32d3d55d791fe294981185 Author: Thomas Huth Date: Mon Sep 9 17:32:56 2013 +0200 KVM: Add documentation for kvm->srcu lock However, this commit is only a documentation

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I found an issue with the image I've been cloning test VMs from which may have affected test results. I've got my own kernel builds set up now and am rerunning the bisect. Will let you know what I find. -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

So, I've had an exciting evening armed with ftrace and a kernel debugger. I'm questioning my own sanity a bit here, but I'm fairly sure at this point that setting ftrace going and loading the kernel debugging module (I'm using kgdboe here because I'm on AWS) stop whatever the issue is from

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Agreed. That means we may have two possible leads for finding the culprit. Are there any seasoned kernel and systemd/udev persons that could look at the changelog and make a guess at which commits we should test? The alternative is to continue to bisect the kernel and also start bisecting udev.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I can't reproduce with the older version of udev either. However, I think this is still a kernel issue, apparently triggered by an interaction with a recent change in udev. Finding the change to udev that causes this could be instructive for figuring out what the underlying kernel issue is, but I

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I've tested 15.10 with different versions, including mainline versions, and I can reproduce the bug on the following version: 4.4.0-040400-generic, 4.2.0-23-generic, 4.0.1-040001-generic, 3.19.8-031908-generic, 3.19.0-43-generic. As the latest 15.04 also gives the symptoms, I suspect it's not

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Thanks a lot for your hard work! I tested it with Amazon EC2 15.10 and it seems that downgrading fixed it. Now testing my Xubuntu Home Server. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Seems also to be fixed on my Xubuntu Home Servers. But I will report back later. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Update: Sadly not fixed for my Home Servers: top - 19:19:36 up 45 min, 1 user, load average: 3,95, 3,81, 3,46 Aufgaben: 175 total, 4 running, 171 sleeping, 0 stopped, 0 zombie %Cpu(s): 1,0 be, 44,8 sy, 1,8 ni, 14,3 un, 37,6 wa, 0,0 hi, 0,5 si, 0,0 st KiB Mem: 1872072 total,

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Issue on AWS t2.micro root@ip-127-0-0-1:~# uname -a Linux ip-127-0-0-1 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux root@ip-127-0-0-1:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=15.10 DISTRIB_CODENAME=wily DISTRIB_DESCRIPTION="Ubuntu

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

This bug kinda makes me curious. I tried to reproduce this issue on a vm but I couldn't get kswapd0 into the 100% cpu state. Isn't it possible to use kernel's lockdep feature to automatically detect the issue if it is an software lock or something like that? -- You received this bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I can also confirm the issue on Amazon AWS t2.nano Linux ip-172-31-5-83 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux after DDing a 512Mb file: top - 18:36:00 up 9 min, 1 user, load average: 1.03, 0.64, 0.30 Tasks: 74 total, 3 running, 71

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I really tried hard DD'ing and filling up RAM but I really can not reproduce the issue under Ubuntu 15.04 on AWS EC2 nano. On my home server boxes that issue also didn't appear on Ubuntu 15.04 either - so I really think it was introduced between 15.04 and 15.10. Linux ip-172-31-7-84

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

For clarity here: I have not tested 15.04 and I can entirely believe that the issue doesn't appear on that version. I'm testing the specified kernel versions on Ubuntu 15.10 and the issue is reliably reproducible. It doesn't seem to depend on whether swap is enabled or how full the RAM is; large

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Could you share the exact commands to reproduce? I tried dd'ing to run out of memory on r2.micro (1GB memory). Please see the attached command logs. Is this comparable with your results? What I find interesting is that on 15.04, it runs out of free memory and starts waiting for I/O (no noteworthy

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

** Attachment added: "15.10-kswapd0.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457/+attachment/4547185/+files/15.10-kswapd0.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Btw, it is very easy to reproduce with t2.nano type of AWS instance. Linux ip-10-0-2-68 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Interesting. I think for now, I should assume that the problem I see is not the same bug. Although the symptoms are very similar with kswapd0 spinning at 100%, swap space not being used, echo 1 > /proc/sys/vm/drop_caches resolving the problem temporarily and the issue first discovered on 15.10 on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 519192aaae38e24d6b32d3d55d791fe294981185 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Sorry for commenting in the middle of your ongoing work, but I struggle matching your findings to my own. Just like Sam and Sean reported, I see the bug on EC2 t2.micro running 15.10. I can reproduce it on t2.small as well, but it takes a little longer as I have to exaust the free memory. I've

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I've found that how easy it is to reproduce varies somewhat - some versions it's triggered at almost any file manipulation, others I've had to copy a couple of large files around at the same time or otherwise increase disk access while moving large files - but I can reproduce it absolutely

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 732e563373ffc57d38a8a3b6d55f2de865182117 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 56aba608257b451f663d25313d5ecae134d5557f The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 59ab5a8f4445699e238c4c46b3da63bb9dc02897 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 98fda169290b3b28c0f2db2b8f02290c13da50ef The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: d7876f1be40a16223a44355740de625849504eb5 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Crashed again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Also see here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1476211 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

This bug is also affecting me on 2 (ident) Xubuntu 15.10 systems: uname -a: ### 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=15.10 DISTRIB_CODENAME=wily DISTRIB_DESCRIPTION="Ubuntu 15.10"

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

That version has also crashed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description:

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 3b7834743f9492e3509930feb4ca47135905e640 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Thanks for testing. I skipped that commit in the bisect, in case it's not related to the bug. I built the next test kernel, up to the following commit: cbbc58d4fdfab1a39a6ac1b41fcb17885952157a The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Same crash on that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: b746f9c7941f227ad582b4f0bc981f3adcbc46b2 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

No bug in that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 72c1253574a1854b0b6f196e24cd0dd08c1ad9b9 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

It's crashing on boot with this version. It's related to paging, so it might be relevant to the issue, so I've attached the full dmesg and here's the actual crash: [3.716345] BUG: unable to handle kernel paging request at 60ffc0002370 [3.720056] IP: []

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Bug is present in this version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description:

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: f080480488028bcc25357f85e8ae54ccc3bb7173 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

No bug in that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: In Progress Bug description: As per

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

** Changed in: linux (Ubuntu) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) ** Changed in: linux (Ubuntu) Status: Triaged => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: ab0169bb5cc4a5c86756dde662087f9d12302eb0 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Bug is present in this version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: Triaged Bug description: As

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: 9073e1a804c3096eda84ee7cbf11d1f174236c75 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

No bug on that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: Triaged Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I built the next test kernel, up to the following commit: e1f56c89b040134add93f686931cc266541d239a The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1518457 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

No bug on that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: Triaged Bug description: As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I started a kernel bisect between v3.12 final and v3.13-rc1. The kernel bisect will require testing of about 7-10 test kernels. I built the first test kernel, up to the following commit: 5cbb3d216e2041700231bcfc383ee5f8b7fc8b74 The test kernel can be downloaded from:

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Thanks for testing, Sam. Could you also test the 3.12 final version, since 3.13-rc1 is the next linear version after 3.12 final. The kernel can be downloaded from: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-trusty/ -- You received this bug notification because you are a member of

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

3.12 final doesn't exhibit the issue either. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in Ubuntu: Triaged Bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue. Can you test the following kernels and report back? We are looking for the first

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Okay, I cloned my server and tried kernel versions. The latest version which does _not_ exhibit the issue is 3.12.51. The first which does is 3.13-rc1. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Pretty much same description here. Started when I upgraded Amazon instance to 15.10. Causing a lot of disruption - available to test also if it helps move us forward. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

** Changed in: linux (Ubuntu) Status: Confirmed => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1518457 Title: kswapd0 100% CPU usage Status in linux package in

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

This was a clean build, so I don't have any information about previous versions unfortunately. (The previous server, which didn't have this issue, was different AWS hardware and the previous Ubuntu version.) I've tested with the latest mainline kernel and this is still occurring. ** Changed in:

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.4

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

same issue here: root@orangepi:/var/log# uname -a Linux orangepi 3.4.39 #2 SMP PREEMPT Mon Oct 12 12:03:03 CEST 2015 armv7l armv7l armv7l GNU/Linux root@orangepi:/var/log# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=15.10 DISTRIB_CODENAME=wily DISTRIB_DESCRIPTION="Ubuntu 15.10" also

<    1   2