[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Brad Figg]

** Tags added: cscc

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Arie Skliarouk]

The guest used lots of memory on NUMA machine. Turned out KSM merged
some of the processes pages causing the particular processes to crawl.

Disabled KSM merge_across_nodes and it solved the problem:
echo 2 > /sys/kernel/mm/ksm/run && sleep 300 && cat 
/sys/kernel/mm/ksm/pages_shared
echo 0 > /sys/kernel/mm/ksm/merge_across_nodes && echo 1 > 
/sys/kernel/mm/ksm/run

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Arie Skliarouk]

I am seeing this bug on a ubuntu 16.04 box running on proxmox 5.1.
Running 4.4.0-72-generic x86_64 guest (linux-image-4.4.0-72-generic).

Under heavy I/O sometimes the guest get stuck for couple of seconds and this 
error is printed:
[538796.529342] hrtimer: interrupt took 37764682 ns

After that the guest becames unbearably slow and only reboot helps.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[to-hiro]

** Changed in: linux (Ubuntu Trusty)
Milestone: None => trusty-updates

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Jim]

This is NOT fixed by 3.13.0-33.58.  It continues to persist even with
3.13.0-65.106 (and 3.13.0-63.103).

I have around 10 VMs running but ONE in particular disconnects from the
network every hour or so.

I had this issue previously but it was initially gone on Ubuntu 14.04
LTS but had come back recently - perhaps some kernel regression ?

dmesg shows

[42524.196629] kvm: zapping shadow pages for mmio generation wraparound
[42538.140013] br0: port 2(vnet0) entered learning state
[42538.268017] br1: port 2(vnet1) entered learning state
[42553.180008] br0: topology change detected, propagating
[42553.180015] br0: port 2(vnet0) entered forwarding state
[42553.308008] br1: topology change detected, propagating
[42553.308014] br1: port 2(vnet1) entered forwarding state

(and NIC connection is gone)

It's not clear if this is just co-incidence or if this is a pointer to
the issue.

This VM is unusual in my VMs becuase it is the only one with 2 NIC
connections to br1 and br0.  All the others connect to just br0.  Those
others work OK.

Happy to try suggestions to track this down.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Re: [Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Thiago Martins]

Why not go with Linux 3.19 ?

You can just install it by running:

sudo apt-get install linux-generic-lts-vivid

It is by far, much more stable than Linux 3.13 (specially the network
stack)... ;-)

On 6 October 2015 at 11:36, Jim  wrote:
> This is NOT fixed by 3.13.0-33.58.  It continues to persist even with
> 3.13.0-65.106 (and 3.13.0-63.103).
>
> I have around 10 VMs running but ONE in particular disconnects from the
> network every hour or so.
>
> I had this issue previously but it was initially gone on Ubuntu 14.04
> LTS but had come back recently - perhaps some kernel regression ?
>
> dmesg shows
>
> [42524.196629] kvm: zapping shadow pages for mmio generation wraparound
> [42538.140013] br0: port 2(vnet0) entered learning state
> [42538.268017] br1: port 2(vnet1) entered learning state
> [42553.180008] br0: topology change detected, propagating
> [42553.180015] br0: port 2(vnet0) entered forwarding state
> [42553.308008] br1: topology change detected, propagating
> [42553.308014] br1: port 2(vnet1) entered forwarding state
>
> (and NIC connection is gone)
>
> It's not clear if this is just co-incidence or if this is a pointer to
> the issue.
>
> This VM is unusual in my VMs becuase it is the only one with 2 NIC
> connections to br1 and br0.  All the others connect to just br0.  Those
> others work OK.
>
> Happy to try suggestions to track this down.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1341195).
> https://bugs.launchpad.net/bugs/1346917
>
> Title:
>   Using KSM on NUMA capable machines can cause KVM guest performance and
>   stability issues
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Chris J Arges]

This could be a different issue. For now can you file a new bug with additional 
information?
Running: 'apport-bug linux' in a terminal would be best as it collects dmesg 
output and package versions.
Any description or examples of how to reproduce would be very helpful in 
tracking this down. Thanks!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Mohammed Naser]

I think this is a different issue you're describing, connectivity
doesn't drop with this but it becomes very flakey, large latency spikes,
etc.  It would never 100% drop

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Dave Chiluk]

For those seeing this issue after 3.13.0-33.58, please ensure that the
virtual machine's host kernel is running 3.13.0-33.58 or newer.  The VM
kernel itself does not matter.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the <
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Mohammed Naser]

I am seeing the issue in 3.13.0-46-generic as well.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Sushitha]

I am seeing the same issue in 3.13.0-46-generic

hrtimer: interrupt took 4352551231 ns

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Jay Janardhan]

I'm seeing the same issue on kernel: 3.13.0-40-generic. Per above
comment the bug is fixed in 3.13.0-33.58 but not included in the later
versions?

Dec  8 16:14:11 vm1 kernel: [  109.084235] random: nonblocking pool is 
initialized
Dec  8 16:49:40 vm1 kernel: [ 2237.458245] hrtimer: interrupt took 42733372 ns

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 3.13.0-33.58

---
linux (3.13.0-33.58) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
- LP: #1349897

  [ Upstream Kernel Changes ]

  * mm: numa: do not automatically migrate KSM pages
- LP: #1346917
  * net: fix UDP tunnel GSO of frag_list GRO packets
- LP: #1331219
  * auditsc: audit_krule mask accesses need bounds checking
- LP: #1347088
  * n_tty: Fix buffer overruns with larger-than-4k pastes
- LP: #1208740
 -- Tim Gardner tim.gard...@canonical.com   Fri, 18 Jul 2014 14:57:50 +

** Changed in: linux (Ubuntu Trusty)
   Status: Fix Committed = Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Launchpad Bug Tracker]

** Branch linked: lp:ubuntu/precise-proposed/linux-lts-trusty

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  Fix Committed

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Chris J Arges]

SRU Patch to Trusty (3.13) submitted to ubuntu-kernel-team ML.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  In Progress

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Tim Gardner]

https://lists.ubuntu.com/archives/kernel-team/2014-July/046805.html

** Changed in: linux (Ubuntu Trusty)
   Status: In Progress = Fix Committed

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  Fix Committed

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1346917] Re: Using KSM on NUMA capable machines can cause KVM guest performance and stability issues

[Chris J Arges]

A test build for this patch is provided here:
http://people.canonical.com/~arges/lp1346917/

For most servers 
linux-image-3.13.0-33-generic_3.13.0-33.58~lp1346917v201407220903_amd64.deb 
should be sufficient; however if you have DKMS packages you may need to install 
linux-headers* packages. The linux-image-extra package has additional modules 
if necessary.
In addition dbgsym and tools packages are provided for debugging.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1346917

Title:
  Using KSM on NUMA capable machines can cause KVM guest performance and
  stability issues

Status in “linux” package in Ubuntu:
  Fix Released
Status in “linux” source package in Trusty:
  In Progress

Bug description:
  [Impact]

  When using KVM on NUMA machines, both Linux and Windows guests can
  exhibit very poor performance and potential crashes. Disabling KSM is
  a known workaround to fix this issue.

  [Fix]

  The following patch fixes the issue in our testing:
  
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=64a9a34e22896dad430e21a28ad8cb00a756fefc

  This patch is present in v3.14-rc1 and onwards.

  [Test Case]

  General test case:
  1) On a NUMA capable machine, setup the machine as a KVM hypervisor
    - lscpu should show more than 1 NUMA node
  2) Install 4 KVM VMs
  3) Run the following in another terminal to ensure that pages_shared and 
pages_sharing is increasing
   - watch 'tail /sys/kernel/mm/ksm/*'
  4) In another terminal run a program that continually pings each node and 
alerts on high latencies

  What we've observed is that in Linux guests, the ping latencies can go
  into the ~2 second range for a few pings, then return back to the 
  1ms range. (This is machine dependent.) In addition, occasionally when
  running this test with Windows guests we observe BSODs during this
  test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp