[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Tags added: cscc -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Opinion Status in linux package in Ubuntu: Invalid Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use /proc/meminfo, etc. Attach the syslog and the /var/log/libvirt/qemu logs. 2. What is running on the host (at
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Changed in: linux (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Opinion Status in linux package in Ubuntu: Invalid Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use /proc/meminfo, etc. Attach the syslog and the /var/log/libvirt/qemu
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
Yes, we have closed the support case on our end at their request. Apparently increasing the reservation ratio has helped. Paulus - Hi! Thanks for the info and clearing up some of my misunderstandings. Great to hear from you and I hope things are going well at OzLabs :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Opinion Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Changed in: ubuntu-power-systems Status: Triaged => Opinion -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Opinion Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use /proc/meminfo, etc. Attach the syslog and the
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Attachment added: "var_log_libvirt_qemu.tar.bz2" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781038/+attachment/5164739/+files/var_log_libvirt_qemu.tar.bz2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Attachment added: "syslog" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781038/+attachment/5164740/+files/syslog -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Attachment added: "meminfo" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781038/+attachment/5164738/+files/meminfo -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
Based on the most recent information we have available to us (2018-05-09): 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use /proc/meminfo, etc. Attach the syslog and the /var/log/libvirt/qemu logs. I am struggling a bit to determine the server model, but I'm uploading the relevant logs. 2. What is running on the host (at least uname -a). Sounds like from comment above like it's an older fix level, so let's get it updated to the curent level (and ensure the problem still exists) before proceeding: There is zero point in trying to figure out whether fixes that are known to exist in 16.04 are in this *particular* build level. Linux apsoscmp-as-a4p 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 20:00:40 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux I don't have any answers for (3); the user has been asked. ** Attachment added: "lspci" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781038/+attachment/5164737/+files/lspci -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Changed in: ubuntu-power-systems Status: New => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is the server model and at least basic config info (I/O cards, firmware level)? Use /proc/meminfo, etc. Attach the syslog and the /var/log/libvirt/qemu
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Changed in: linux (Ubuntu) Importance: Undecided => Critical ** Changed in: linux (Ubuntu) Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) => Canonical Kernel Team (canonical-kernel-team) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to get started: 1. What is
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
Hi, This came up in the context of a customer issue. I have asked them if we can share anonymised data here, and I will pass on any response. >From my analysis of the code while working the case, it would seem that you could reproduce this by spinning up and tearing down VMs of varying memory sizes in order to fragment the CMA. It looks like PCI pass- through would exacerbate the issue, although I don't believe this was a factor in this instance. I wonder if this is fully 'solvable' per se - with memory overcommit it should be easy to simply run out of CMA space - but it should be possible to at least print much more helpful information either from the kernel or from qemu. Regards, Daniel -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
** Also affects: ubuntu-power-systems Importance: Undecided Status: New ** Changed in: ubuntu-power-systems Importance: Undecided => Critical ** Changed in: ubuntu-power-systems Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team) ** Tags added: triage-g -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but,
[Kernel-packages] [Bug 1781038] Re: KVM guest hash page table failed to allocate contiguous memory (CMA)
Hello Canonical, I've subscribed Daniel to this bug for his help getting additional information on the situation he was working. I expect he will add more folks from his end if necessary. Lastly, if you feel the bug should be marked as private for this discussion, please feel free to do that. Thanks. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1781038 Title: KVM guest hash page table failed to allocate contiguous memory (CMA) Status in linux package in Ubuntu: New Bug description: Per an email forwarded within IBM, we wish to use this Launchpad bug to work on the technical discussion with the Canonical development folks and the IBM KVM and kernel team surrounding the analysis made by Daniel Axtens of Canonical for the customer issue raised in Case #00177825. The only statement at the moment by the KVM team was that there were various issues associated with CMA fragmentation causing issues with KVM guests. However, as mentioned, this bug is to allow the dialog amongst all the developers to see what can be done to help alleviate the situation or understand the root cause further. Please also note that we should not be attaching customer data to this bug. If that is necessary then we expect Canonical to help provide a controlled environment for reviewing that data so we avoid any privacy issues (e.g. for GDPR compliance). Here is the email from Daniel: I have looked at the sosreport you uploaded. Here is my analysis so far. Virtualisation on powerpc has some special requirements. To start a guest on a powerpc host, you need to allocate a contiguous area of memory to hold the guest's hash page table (HPT, or HTAB, depending on which document you look at). The HPT is required to track and manage guest memory. Your error reports show qemu asking the kernel to allocate an HTAB, and the kernel reporting that it had insufficient memory to do so. The required memory for the HPT scales with the guest memory size - it should be about 1/128th of guest memory, so for a 16GB guest, that's 128MB. However, the HPT has to be allocated as a single contiguous memory region. (This is in contrast to regular guest memory, which is not required to be contiguous from the host point of view.) The kernel keeps a special contiguous memory area (CMA) for these purposes, and keeps track of the total amounts in use and still available. These are shown in /proc/meminfo. From the system that ran the sosreport, we see: CmaTotal: 26853376 kB CmaFree: 4024448 kB So there is a total of about 25GB of CMA, of which about 3.8GB remain. This is obviously more than 128MB: - It's very possible that between the error and the sosreport, more contiguous memory became available. This would match the intermittent nature of the issue. - It also might be that the failure was due to fragmentation of memory in the CMA pool. That is, there might be more than 128MB, but it might all be in chunks that are smaller than 128MB, or which don't have the required alignment for a HPT. Given that the system's uptime was 112 days when the sosreport was generated, it would be unsurprising if fragmentation had occurred! (Relatedly - you're running 4.4.0-109, which does not have the Spectre and Meltdown fixes.) This issue has come up before - both in a public Canonical-IBM synchronised bug report[1], and with Red Hat[2]. It appears that there is some work within IBM to address this, but it seems to have stalled. I will get in touch with the IBM powerpc kernel team on their public mailing list and ask about the status. I will keep you updated. In the mean time, I have a potential solution/workaround. By default, 5% of memory is reserved for CMA (kernel source: arch/powerpc/kvm/book3s_hv_builtin.c, kvm_cma_resv_ratio). You can increase this with a boot parameter, so for example to reserve 10%, you could boot with kvm_cma_resv_ratio=10. This can be set in petitboot. This should significantly reduce the incidence of this issue - perhaps eliminating it entirely - at the cost of locking away more of the system's memory. You would need to experiment to determine the optimal value. Perhaps given that you are seeing the problem only intermittently, a ratio of 7% would be sufficient - that would give you ~35GB of CMA. Please let me know if testing this setting would be an option for you. Please also let me know if you require further information on setting boot parameters with Petitboot. Regards, Daniel [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1632045 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1304300 Before we go any further, let's get the basic info here. Apparently there was a sosreport somewhere else, and a link would be good, but, here's what we need here -- at least -- to
