from:"Gregg King"

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

2016-09-03 Thread Gregg King

Did you try this from Rasmussen up above?

sudo touch /etc/udev/rules.d/40-vm-hotadd.rules
reboot

That fixed it for me on EC2

On Saturday, 3 September 2016, Andy Robertson 
wrote:

> All of my t2.micro & nano instances are affected by this in AWS EC2
> after upgrading to Ubuntu16.
>
> doing "echo 1 > /proc/sys/vm/drop_caches" (also tried echo 3) works for
> a short period of time, but it comes back within a few minutes.
>
> I moved a couple instances over to f1.micro on GCP / GCE (1 vCPU, 0.6 GB
> memory) and the problem seems to have gone away.  I can't do this with
> all of my instances yet though so a fix in AWS would be nice.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1518457
>
> Title:
>   kswapd0 100% CPU usage
>
> Status in Linux:
>   Unknown
> Status in linux package in Ubuntu:
>   Confirmed
>
> Bug description:
>   As per bug 721896 and various others:
>
>   I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
>   Occasionally (about once a day), kswapd0 falls into a busy loop and
>   spins on 100% CPU usage indefinitely. This can be provoked by
>   copying/writing large files (e.g. dding a 256MB file), but it happens
>   occasionally otherwise. System memory usage (not including
>   buffers/caches) currently sits at 36%, which is typical[1]. Initially
>   I had no swap space configured; I've since tried enabling a 256MB swap
>   file, but the problem continues to occur and no swap space is used.
>   The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.
>
>   Happy to provide further information/take further debugging actions.
>
>
>   [1] Full output from `free`:
>total   used   free sharedbuffers cached
>   Mem:   1014936 483448 531488  28556   9756 112700
>   -/+ buffers/cache: 360992 653944
>   Swap:   262140  0 262140
>
>   ProblemType: Bug
>   DistroRelease: Ubuntu 15.10
>   Package: linux-image-4.2.0-18-generic 4.2.0-18.22
>   ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
>   Uname: Linux 4.2.0-18-generic x86_64
>   AlsaDevices:
>total 0
>crw-rw 1 root audio 116,  1 Nov 19 19:40 seq
>crw-rw 1 root audio 116, 33 Nov 19 19:40 timer
>   AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
>   ApportVersion: 2.19.1-0ubuntu5
>   Architecture: amd64
>   ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
>   AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
> '/dev/snd/timer'] failed with exit code 1:
>   CRDA: N/A
>   Date: Fri Nov 20 20:44:30 2015
>   Ec2AMI: ami-1c552a76
>   Ec2AMIManifest: (unknown)
>   Ec2AvailabilityZone: us-east-1d
>   Ec2InstanceType: t2.micro
>   Ec2Kernel: unavailable
>   Ec2Ramdisk: unavailable
>   IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
>   Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to
> initialize libusb: -99
>   MachineType: Xen HVM domU
>   PciMultimedia:
>
>   ProcEnviron:
>TERM=screen
>PATH=(custom, no user)
>LANG=en_US.UTF-8
>SHELL=/bin/bash
>   ProcFB: 0 xen
>   ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic
> root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1
> console=ttyS0 net.ifnames=0
>   RelatedPackageVersions:
>linux-restricted-modules-4.2.0-18-generic N/A
>linux-backports-modules-4.2.0-18-generic  N/A
>linux-firmwareN/A
>   RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
>   SourcePackage: linux
>   UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
>   UpgradeStatus: No upgrade log present (probably fresh install)
>   dmi.bios.date: 05/06/2015
>   dmi.bios.vendor: Xen
>   dmi.bios.version: 4.2.amazon
>   dmi.chassis.type: 1
>   dmi.chassis.vendor: Xen
>   dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd05/
> 06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
>   dmi.product.name: HVM domU
>   dmi.product.version: 4.2.amazon
>   dmi.sys.vendor: Xen
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linux/+bug/1518457/+subscriptions
>

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1518457

Title:
  kswapd0 100% CPU usage

Status in Linux:
  Unknown
Status in linux package in Ubuntu:
  Confirmed

Bug description:
  As per bug 721896 and various others:

  I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
  Occasionally (about once a day), kswapd0 falls into a busy loop and
  spins on 100% CPU usage indefinitely. This can be provoked by
  copying/writing large files (e.g. dding a 256MB file), but it happens
  occasionally otherwise. System memory usage (not including
  buffers/caches) currently sits at 36%, which is typical[1]. Initially
  I had no swap

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

2016-06-19 Thread Gregg King

Thank you so much Rasmus!

Your solution worked for me:
sudo touch /etc/udev/rules.d/40-vm-hotadd.rules
reboot

I could always trigger the CPU usage bug with this:
stress --cpu 8 --io 4 --vm 7 --vm-bytes 128M --vm-hang 3 --timeout 60s

If running it once didn't work, a second run would do it. Now I've run it
over and over and kswap hits 1-2% during the stress test and drops back
down as soon as it ends.

Like Rasmus says, it's an AWS instance which runs on Xen.


On 17 June 2016 at 07:37, Rasmus Larsen  wrote:

> I had this issue too on AWS.
>
> In my case, it was the udev rule for vm-hotadd and the fix as mentioned
> previously basically came down to "touch /etc/udev/rules.d/40-vm-
> hotadd.rules" which effectively disables the /lib/udev/rules.d/40-vm-
> hotadd.rules file (after a reboot).
>
> The udev rule basically seems to only be active for Xen or Hyper-V and
> while it seems the Hyper-V stuff was also present in previous versions,
> the Xen stuff seems to be introduced in 15.10 or newer.
>
> So if you're seeing this issue on anything running on Xen, including
> AWS, try:
>
> touch /etc/udev/rules.d/40-vm-hotadd.rules
> reboot
>
> This is probably a bug in Xen or a bug in the kernel.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1518457
>
> Title:
>   kswapd0 100% CPU usage
>
> Status in Linux:
>   Unknown
> Status in linux package in Ubuntu:
>   Confirmed
>
> Bug description:
>   As per bug 721896 and various others:
>
>   I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
>   Occasionally (about once a day), kswapd0 falls into a busy loop and
>   spins on 100% CPU usage indefinitely. This can be provoked by
>   copying/writing large files (e.g. dding a 256MB file), but it happens
>   occasionally otherwise. System memory usage (not including
>   buffers/caches) currently sits at 36%, which is typical[1]. Initially
>   I had no swap space configured; I've since tried enabling a 256MB swap
>   file, but the problem continues to occur and no swap space is used.
>   The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.
>
>   Happy to provide further information/take further debugging actions.
>
>
>   [1] Full output from `free`:
>total   used   free sharedbuffers cached
>   Mem:   1014936 483448 531488  28556   9756 112700
>   -/+ buffers/cache: 360992 653944
>   Swap:   262140  0 262140
>
>   ProblemType: Bug
>   DistroRelease: Ubuntu 15.10
>   Package: linux-image-4.2.0-18-generic 4.2.0-18.22
>   ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
>   Uname: Linux 4.2.0-18-generic x86_64
>   AlsaDevices:
>total 0
>crw-rw 1 root audio 116,  1 Nov 19 19:40 seq
>crw-rw 1 root audio 116, 33 Nov 19 19:40 timer
>   AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
>   ApportVersion: 2.19.1-0ubuntu5
>   Architecture: amd64
>   ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
>   AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
> '/dev/snd/timer'] failed with exit code 1:
>   CRDA: N/A
>   Date: Fri Nov 20 20:44:30 2015
>   Ec2AMI: ami-1c552a76
>   Ec2AMIManifest: (unknown)
>   Ec2AvailabilityZone: us-east-1d
>   Ec2InstanceType: t2.micro
>   Ec2Kernel: unavailable
>   Ec2Ramdisk: unavailable
>   IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
>   Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to
> initialize libusb: -99
>   MachineType: Xen HVM domU
>   PciMultimedia:
>
>   ProcEnviron:
>TERM=screen
>PATH=(custom, no user)
>LANG=en_US.UTF-8
>SHELL=/bin/bash
>   ProcFB: 0 xen
>   ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic
> root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1
> console=ttyS0 net.ifnames=0
>   RelatedPackageVersions:
>linux-restricted-modules-4.2.0-18-generic N/A
>linux-backports-modules-4.2.0-18-generic  N/A
>linux-firmwareN/A
>   RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
>   SourcePackage: linux
>   UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
>   UpgradeStatus: No upgrade log present (probably fresh install)
>   dmi.bios.date: 05/06/2015
>   dmi.bios.vendor: Xen
>   dmi.bios.version: 4.2.amazon
>   dmi.chassis.type: 1
>   dmi.chassis.vendor: Xen
>   dmi.modalias:
> dmi:bvnXen:bvr4.2.amazon:bd05/06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
>   dmi.product.name: HVM domU
>   dmi.product.version: 4.2.amazon
>   dmi.sys.vendor: Xen
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linux/+bug/1518457/+subscriptions
>

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1518457

Title:
  kswapd0 100% CPU usage

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

2016-06-04 Thread Gregg King

I've had the issue mostly when running with no swap. I added 1GB swap
file and it fixed it on the machine with less memory usage, but the
machine with the database (using more memory most of the time) still has
the bug occur about once a day.

Ubuntu 16.04 for both, not happening on 14.04.

Sent from my iPhone

> On Jun 4, 2016, at 5:11 PM, Jonathan Vargas <1518...@bugs.launchpad.net> 
> wrote:
> 
> I face the same issue, running a t2.micro (1 GB) for a Nginx/Ruby
> application with really low demand. The kswapd0 process takes the CPU to
> 100% and the overall system performance is downgraded.
> 
> Linux server 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> 
> To reproduce it, I just create a 1GB file in the filesystem using "dd":
> 
> if=/dev/zero of=temp bs=1k count=1024k
> 
> I created a swap file to solve the issue but it continues. After
> dropping the page cache with:
> 
> echo 3 > /proc/sys/vm/drop_caches
> 
> the issue is solved temporarily.
> 
> 
> ** Attachment added: "Selection_320.png"
>   
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457/+attachment/4677030/+files/Selection_320.png
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1518457
> 
> Title:
>  kswapd0 100% CPU usage
> 
> Status in Linux:
>  Unknown
> Status in linux package in Ubuntu:
>  Confirmed
> 
> Bug description:
>  As per bug 721896 and various others:
> 
>  I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
>  Occasionally (about once a day), kswapd0 falls into a busy loop and
>  spins on 100% CPU usage indefinitely. This can be provoked by
>  copying/writing large files (e.g. dding a 256MB file), but it happens
>  occasionally otherwise. System memory usage (not including
>  buffers/caches) currently sits at 36%, which is typical[1]. Initially
>  I had no swap space configured; I've since tried enabling a 256MB swap
>  file, but the problem continues to occur and no swap space is used.
>  The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.
> 
>  Happy to provide further information/take further debugging actions.
> 
> 
>  [1] Full output from `free`:
>   total   used   free sharedbuffers cached
>  Mem:   1014936 483448 531488  28556   9756 112700
>  -/+ buffers/cache: 360992 653944
>  Swap:   262140  0 262140
> 
>  ProblemType: Bug
>  DistroRelease: Ubuntu 15.10
>  Package: linux-image-4.2.0-18-generic 4.2.0-18.22
>  ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
>  Uname: Linux 4.2.0-18-generic x86_64
>  AlsaDevices:
>   total 0
>   crw-rw 1 root audio 116,  1 Nov 19 19:40 seq
>   crw-rw 1 root audio 116, 33 Nov 19 19:40 timer
>  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
>  ApportVersion: 2.19.1-0ubuntu5
>  Architecture: amd64
>  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
>  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
> '/dev/snd/timer'] failed with exit code 1:
>  CRDA: N/A
>  Date: Fri Nov 20 20:44:30 2015
>  Ec2AMI: ami-1c552a76
>  Ec2AMIManifest: (unknown)
>  Ec2AvailabilityZone: us-east-1d
>  Ec2InstanceType: t2.micro
>  Ec2Kernel: unavailable
>  Ec2Ramdisk: unavailable
>  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
>  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to 
> initialize libusb: -99
>  MachineType: Xen HVM domU
>  PciMultimedia:
> 
>  ProcEnviron:
>   TERM=screen
>   PATH=(custom, no user)
>   LANG=en_US.UTF-8
>   SHELL=/bin/bash
>  ProcFB: 0 xen
>  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic 
> root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1 console=ttyS0 
> net.ifnames=0
>  RelatedPackageVersions:
>   linux-restricted-modules-4.2.0-18-generic N/A
>   linux-backports-modules-4.2.0-18-generic  N/A
>   linux-firmwareN/A
>  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
>  SourcePackage: linux
>  UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
>  UpgradeStatus: No upgrade log present (probably fresh install)
>  dmi.bios.date: 05/06/2015
>  dmi.bios.vendor: Xen
>  dmi.bios.version: 4.2.amazon
>  dmi.chassis.type: 1
>  dmi.chassis.vendor: Xen
>  dmi.modalias: 
> dmi:bvnXen:bvr4.2.amazon:bd05/06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
>  dmi.product.name: HVM domU
>  dmi.product.version: 4.2.amazon
>  dmi.sys.vendor: Xen
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linux/+bug/1518457/+subscriptions

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1518457

Title:
  kswapd0 100% CPU usage

Status in Linux:
  Unknown
Status in linux package in Ubuntu:
  Confirmed

Bug description:
  As per bug

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

2016-05-11 Thread Gregg King

I'm also seeing this on 16.04 on an AWS nano (~0.5GB RAM). More than
happy to help troubleshoot this if people let me know what info is
needed.

Here's some basic info from my machine:

$ uname -a
4.4.0-22-generic #39-Ubuntu SMP Thu May 5 16:53:32 UTC 2016 x86_64 x86_64 
x86_64 GNU/Linux

$ free -m
  totalusedfree  shared  buff/cache   available
Mem:486  73 140  20 272 362
Swap: 0   0   0


$ top
top - 01:44:48 up 3 days,  2:34,  1 user,  load average: 1.00, 1.01, 1.05
Tasks: 131 total,   2 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 90.9 sy,  0.0 ni,  9.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   498480 total,   143612 free,75852 used,   279016 buff/cache
KiB Swap:0 total,0 free,0 used.   370732 avail Mem

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
   29 root  20   0   0  0  0 R 99.9  0.0   4066:36 kswapd0
1 root  20   0   37908   6028   4048 S  0.0  1.2   0:08.79 systemd
2 root  20   0   0  0  0 S  0.0  0.0   0:00.00 kthreadd
- CUT THE REST -

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1518457

Title:
  kswapd0 100% CPU usage

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  As per bug 721896 and various others:

  I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
  Occasionally (about once a day), kswapd0 falls into a busy loop and
  spins on 100% CPU usage indefinitely. This can be provoked by
  copying/writing large files (e.g. dding a 256MB file), but it happens
  occasionally otherwise. System memory usage (not including
  buffers/caches) currently sits at 36%, which is typical[1]. Initially
  I had no swap space configured; I've since tried enabling a 256MB swap
  file, but the problem continues to occur and no swap space is used.
  The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.

  Happy to provide further information/take further debugging actions.

  
  [1] Full output from `free`:
   total   used   free sharedbuffers cached
  Mem:   1014936 483448 531488  28556   9756 112700
  -/+ buffers/cache: 360992 653944
  Swap:   262140  0 262140

  ProblemType: Bug
  DistroRelease: Ubuntu 15.10
  Package: linux-image-4.2.0-18-generic 4.2.0-18.22
  ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
  Uname: Linux 4.2.0-18-generic x86_64
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Nov 19 19:40 seq
   crw-rw 1 root audio 116, 33 Nov 19 19:40 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.19.1-0ubuntu5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  Date: Fri Nov 20 20:44:30 2015
  Ec2AMI: ami-1c552a76
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1d
  Ec2InstanceType: t2.micro
  Ec2Kernel: unavailable
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize 
libusb: -99
  MachineType: Xen HVM domU
  PciMultimedia:
   
  ProcEnviron:
   TERM=screen
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 xen
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic 
root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1 console=ttyS0 
net.ifnames=0
  RelatedPackageVersions:
   linux-restricted-modules-4.2.0-18-generic N/A
   linux-backports-modules-4.2.0-18-generic  N/A
   linux-firmwareN/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 05/06/2015
  dmi.bios.vendor: Xen
  dmi.bios.version: 4.2.amazon
  dmi.chassis.type: 1
  dmi.chassis.vendor: Xen
  dmi.modalias: 
dmi:bvnXen:bvr4.2.amazon:bd05/06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
  dmi.product.name: HVM domU
  dmi.product.version: 4.2.amazon
  dmi.sys.vendor: Xen

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

Re: [Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

[Kernel-packages] [Bug 1518457] Re: kswapd0 100% CPU usage

4 matches

Site Navigation

Mail list logo

Footer information