Hello Canonical,

I've subscribed Daniel to this bug for his help getting additional
information on the situation he was working. I expect he will add more
folks from his end if necessary.

Lastly, if you feel the bug should be marked as private for this
discussion, please feel free to do that. Thanks.

You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.

  KVM guest hash page table failed to allocate contiguous memory (CMA)

Status in linux package in Ubuntu:

Bug description:
  Per an email forwarded within IBM, we wish to use this Launchpad bug
  to work on the technical discussion with the Canonical development
  folks and the IBM KVM and kernel team surrounding the analysis made by
  Daniel Axtens of Canonical for the customer issue raised in Case

  The only statement at the moment by the KVM team was that there were
  various issues associated with CMA fragmentation causing issues with
  KVM guests. However, as mentioned, this bug is to allow the dialog
  amongst all the developers to see what can be done to help alleviate
  the situation or understand the root cause further.

  Please also note that we should not be attaching customer data to this
  bug. If that is necessary then we expect Canonical to help provide a
  controlled environment for reviewing that data so we avoid any privacy
  issues (e.g. for GDPR compliance).

  Here is the email from Daniel:

  I have looked at the sosreport you uploaded. Here is my analysis so

  Virtualisation on powerpc has some special requirements. To start a
  guest on a powerpc host, you need to allocate a contiguous area of
  memory to hold the guest's hash page table (HPT, or HTAB, depending on
  which document you look at). The HPT is required to track and manage
  guest memory.

  Your error reports show qemu asking the kernel to allocate an HTAB,
  and the kernel reporting that it had insufficient memory to do so. The
  required memory for the HPT scales with the guest memory size - it
  should be about 1/128th of guest memory, so for a 16GB guest, that's
  128MB. However, the HPT has to be allocated as a single contiguous
  memory region. (This is in contrast to regular guest memory, which is
  not required to be contiguous from the host point of view.)

  The kernel keeps a special contiguous memory area (CMA) for these
  purposes, and keeps track of the total amounts in use and still
  available. These are shown in /proc/meminfo. From the system that ran
  the sosreport, we see:

  CmaTotal: 26853376 kB
  CmaFree: 4024448 kB

  So there is a total of about 25GB of CMA, of which about 3.8GB remain.
  This is obviously more than 128MB:

  - It's very possible that between the error and the sosreport, more
  contiguous memory became available. This would match the intermittent
  nature of the issue.

  - It also might be that the failure was due to fragmentation of memory
  in the CMA pool. That is, there might be more than 128MB, but it might
  all be in chunks that are smaller than 128MB, or which don't have the
  required alignment for a HPT.

  Given that the system's uptime was 112 days when the sosreport was
  generated, it would be unsurprising if fragmentation had occurred!
  (Relatedly - you're running 4.4.0-109, which does not have the Spectre
  and Meltdown fixes.)

  This issue has come up before - both in a public Canonical-IBM
  synchronised bug report[1], and with Red Hat[2]. It appears that there
  is some work within IBM to address this, but it seems to have stalled.
  I will get in touch with the IBM powerpc kernel team on their public
  mailing list and ask about the status. I will keep you updated.

  In the mean time, I have a potential solution/workaround. By default,
  5% of memory is reserved for CMA (kernel source:
  arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can
  increase this with a boot parameter, so for example to reserve 10%,
  you could boot with kvm_cma_resv_ratio=10. This can be set in
  petitboot. This should significantly reduce the incidence of this
  issue - perhaps eliminating it entirely - at the cost of locking away
  more of the system's memory. You would need to experiment to determine
  the optimal value. Perhaps given that you are seeing the problem only
  intermittently, a ratio of 7% would be sufficient - that would give
  you ~35GB of CMA.

  Please let me know if testing this setting would be an option for you.
  Please also let me know if you require further information on setting
  boot parameters with Petitboot.


  [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045
  [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300

  Before we go any further, let's get the basic info here. Apparently
  there was a sosreport somewhere else, and a link would be good, but,
  here's what we need here -- at least -- to get started:

  1. What is the server model and at least basic config info (I/O cards,
  firmware level)? Use /proc/meminfo, etc.  Attach the syslog and the
  /var/log/libvirt/qemu logs.

  2. What is running on the host (at least uname -a). Sounds like from
  comment above like it's an older fix level, so let's get it updated to
  the curent level (and ensure the problem still exists) before
  proceeding: There is zero point in trying to figure out whether fixes
  that are known to exist in 16.04 are in this *particular* build level.

  3. What is running on the guests? The exact same OS level? Please
  attach XML (from virsh dumpxml) for each guest running on the system
  when the failure occurs (and make a note of which one is from the
  failing guest). If we are 100% sure that, excepting unique IDs &
  filenames, the XMLs are identical, then don't attach duplicates.

  4. Anything else you think we should know.

To manage notifications about this bug go to:

Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to