[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after updating kernel from

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

This bug was fixed in the package linux - 4.13.0-38.43 --- linux (4.13.0-38.43) artful; urgency=medium * linux: 4.13.0-38.43 -proposed tracker (LP: #1755762) * Servers going OOM after updating kernel from 4.10 to 4.13 (LP: #1748408) - i40e: Fix memory leak related filter

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

For how long do the Xenial HWE kernels stay in the "proposed" ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after updating kernel from 4.10 to 4.13

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Proposed kernels show the same improved behaviour as the earlier test kernels. ** Tags removed: verification-needed-artful ** Tags added: verification-done-artful -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Running the -proposed kernel on two machines now, will provide the results in a couple of days. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

** Changed in: linux (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after updating kernel from 4.10

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Together with the new Artful kernel there was also a new HWE kernel that is based on the new Artful kernel (4.13.0-38.43~16.04.1). Verification can be done with that kernel as well. Just the automatically generated messages are for the base kernels where the patch was applied to. The HWE kernel is

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

@Stefan: I haven't reproduced the issue on Artful and I don't have an environment to do so. The original issue is for the HWE kernel on Xenial and only for that I can perform verification. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- artful' to 'verification-done-artful'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after updating kernel

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

SRU request submitted: https://lists.ubuntu.com/archives/kernel-team/2018-March/090701.html ** Description changed: - We are seeing this on multiple servers after upgrading from previous - 4.10 series HWE kernels to the new 4.13 HWE series. With the new kernel, - free memory is continously

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

** Changed in: linux (Ubuntu) Status: Triaged => In Progress ** Changed in: linux (Ubuntu Artful) Status: Triaged => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

The slow leak will probably be tolerable for the time being, having those two patches added to the kernel would surely be a pretty valuable step that I think should be done now. My target still is Xenial with the hwe kernel, though. If you need to go via Artful to fix that, well, go ahead. --

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Do you think an Artful SRU request should be sent for commits 2b9478 and 62b4c66? Or would you like to investigate the slow memory leak further? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

The test kernel solves the issue in the same way as my own kernel earlier, i.e. we still seem to have a very slow running memory leak with this kernel. I'm also seeing this slow leak when I replace the in-tree i40e driver by an upstream version (2.3.4), so either it is unrelated or contained in

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

I built a test kernel with commits 2b9478 and 62b4c66. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1748408 Can you test this kernel and see if it resolves this bug? Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

After running for a couple of days, it seems that we are still seeing the slow memory leak similar to what was noticed in >= 4.14 earlier with the patched kernel. But it won't be possible for me to bisect at that rate. @Joseph: Getting a patched current 4.13 still would be nice, getting

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

I have a significant memory leak after upgrading from previous 4.10 series HWE kernels to the new 4.13 HWE series for Ubuntu 16.04 server with Ethernet controller Intel X710 for 10GbE SFP+ # dmesg | grep i40e [1.625565] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

O.k., confirming that this series of patches fixes the issue: ~/linux$ git log --oneline|head -3 bc6d6fd2f916 i40e: Add programming descriptors to cleaned_count 69949b3bd674 i40e: Fix memory leak related filter programming status b32038eb34ee UBUNTU: Ubuntu-4.13.0-32.35 Can you build the same

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Reading the thread further, we seem to need two patches, see https://www.spinics.net/lists/netdev/msg462051.html, so I'm going to add bc6d6fd2f916a0794ae4c44b28e14e2d172e05e0 into the build, too. Will try that on top of b32038eb34ee42fd8056f99f88652270f6667996 (tag: Ubuntu-4.13.0-32.35). I

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

I built a test kernel with commit 2b9478ffc550f1. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1748408 Can you test this kernel and see if it resolves this bug? Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

A colleague found that this seems to be a known issue: https://www.spinics.net/lists/netdev/msg458258.html and the fix should be https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/drivers/net/ethernet/intel/i40e/i40e_txrx.c?id=2b9478ffc550f17c6cd8c69057234e91150f5972 I will

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Sorry for the delay, bisecting took longer than planned, but I now have the result: 6964e53f55837b0c49ed60d36656d2e0ee4fc27b is the first bad commit commit 6964e53f55837b0c49ed60d36656d2e0ee4fc27b Author: Jacob Keller Date: Mon Jun 12 15:38:36 2017 -0700 i40e:

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

@Dr. Jens Harbott, Just let me know if you need assistance with the bisect between 4.11 and 4.12. I can build the kernels for you. I would say the next step would be to test the 4.12 release candidates to narrow down the issue further. 4.12-rc1 is available here:

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

** Tags added: kernel-bug-exists-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title: Servers going OOM after updating kernel from 4.10 to 4.13 Status in linux

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Oh and I have also experienced random, unexplained OOM crashing of squid process recently on two separate machines (one i386 and one amd64) which seem to coincide in time with the upgrade from kernel 4.10 to 4.13 -- You received this bug notification because you are a member of Kernel Packages,

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Hi guys. I'm experiencing problems with 4.13.0-36 on a cloud server (x64) with dynamic ram management. The server is provisioned to use up to 12GB of RAM, but it got so bad that only 1GB was visible, causing everything to a halt while swap usage went through the roof. Could it be related? And

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

So here are the first results: 4.11.0-041100-generic #201705041534 - not affected 4.12.0-041200-generic #201707022031 - affected 4.13.0-041300-generic #201709031731 - affected 4.13.16-041316-generic #201711240901 - affected Results for newer kernels are not so clear, they do not fail as fast as

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Thanks for the update. I requested testing of the mainline kernel to see if there is a commit in mainline that fixes the bug, which we could backport back to 4.13. If the bug is not fixed in mailine, we can perform a kernel bisect to identify the commit that introduced this regression. -- You

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Ok, nevermind the aufs issue, I got that resolved. Should have some results with mainline kernels in a couple of days. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1748408 Title:

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

@Joseph: I did test 4.15.2, but some things are failing, in particular docker because of lacking AUFS support, so I need to build a kernel myself I guess, which will take a bit. Also note that I'm seeing this on Xenial machines, didn't test with Artful. We used to run them with the 4.10 HWE

[Kernel-packages] [Bug 1748408] Re: Servers going OOM after updating kernel from 4.10 to 4.13

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.15 kernel[0]. Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15 ** Package changed: linux-hwe (Ubuntu) => linux (Ubuntu)