Re: Memory locking limit and zero-copy migrations
On Fri, Aug 19, 2022 at 08:01:58 +0200, Milan Zamazal wrote: > Fangge Jin writes: > > > On Fri, Aug 19, 2022 at 4:08 AM Milan Zamazal wrote: > > > >> > Not sure whether you already know this, but I had a hard time > >> > differentiating the two concepts: > >> > 1. memlock hard limit(shown by prlimit): the hard limit for locked host > >> > memory > >> > 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use > >> host > >> > memory, this memory can be swapped out. > >> > >> No, I didn't know it, thank you for pointing this out. Indeed, 2. is > >> what both the libvirt and kernel documentation seem to say, although not > >> so clearly. > >> > >> But when I add with to the domain XML and then > >> start the VM, I can see the limit shown by `prlimit -l' is increased > >> accordingly. This is good for my use case, but does it match what you > >> say about the two concepts? > > > > memtune->hard_limit(hard limit of in-use memory) actually takes effect via > > cgroup, > > you can check the value by: > > # virsh memtune uefi1 > > hard_limit : 134217728 > > soft_limit : unlimited > > swap_hard_limit: unlimited > > # cat > > /sys/fs/cgroup/memory/machine.slice/machine-qemu\\x2d6\\x2duefi1.scope/libvirt/memory.limit_in_bytes > > > > 137438953472 > > > > When vm starts with memtune->hard_limit set in domain XML, memlock > > hard limit( hard_limit of locked memory, shown by 'prlimit -l')will be > > set to the value of memtune->hard_limit. This's probably because > > memlock hard limit must be less than memtune->hard_limit. > > Well, increasing the memlock limit to keep it within memtune->hard_limit > wouldn't make much sense, but thank you for confirming that setting > memtune->hard_limit adjusts both the limits to the requested value. Right, whenever libvirt needs to set a memory locking limit for zero copy migration it uses memtune->hard_limit if it is set (RDMA migration does the same), otherwise the configured memory amount for the domain is used. Anyway, unless you have a specific reason for setting the hard_limit, it's usually better to leave it unset as the overall memory consumption of a domain including the memory consumed by QEMU is hard to predict and you would risk the domain to be killed unexpectedly by the kernel when the limit is reached. Jirka
Re: Memory locking limit and zero-copy migrations
Fangge Jin writes: > On Fri, Aug 19, 2022 at 4:08 AM Milan Zamazal wrote: > >> > Not sure whether you already know this, but I had a hard time >> > differentiating the two concepts: >> > 1. memlock hard limit(shown by prlimit): the hard limit for locked host >> > memory >> > 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use >> host >> > memory, this memory can be swapped out. >> >> No, I didn't know it, thank you for pointing this out. Indeed, 2. is >> what both the libvirt and kernel documentation seem to say, although not >> so clearly. >> >> But when I add with to the domain XML and then >> start the VM, I can see the limit shown by `prlimit -l' is increased >> accordingly. This is good for my use case, but does it match what you >> say about the two concepts? > > memtune->hard_limit(hard limit of in-use memory) actually takes effect via > cgroup, > you can check the value by: > # virsh memtune uefi1 > hard_limit : 134217728 > soft_limit : unlimited > swap_hard_limit: unlimited > # cat > /sys/fs/cgroup/memory/machine.slice/machine-qemu\\x2d6\\x2duefi1.scope/libvirt/memory.limit_in_bytes > > 137438953472 > > When vm starts with memtune->hard_limit set in domain XML, memlock > hard limit( hard_limit of locked memory, shown by 'prlimit -l')will be > set to the value of memtune->hard_limit. This's probably because > memlock hard limit must be less than memtune->hard_limit. Well, increasing the memlock limit to keep it within memtune->hard_limit wouldn't make much sense, but thank you for confirming that setting memtune->hard_limit adjusts both the limits to the requested value.
Re: Memory locking limit and zero-copy migrations
On Fri, Aug 19, 2022 at 4:08 AM Milan Zamazal wrote: > > Not sure whether you already know this, but I had a hard time > > differentiating the two concepts: > > 1. memlock hard limit(shown by prlimit): the hard limit for locked host > > memory > > 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use > host > > memory, this memory can be swapped out. > > No, I didn't know it, thank you for pointing this out. Indeed, 2. is > what both the libvirt and kernel documentation seem to say, although not > so clearly. > > But when I add with to the domain XML and then > start the VM, I can see the limit shown by `prlimit -l' is increased > accordingly. This is good for my use case, but does it match what you > say about the two concepts? memtune->hard_limit(hard limit of in-use memory) actually takes effect via cgroup, you can check the value by: # virsh memtune uefi1 hard_limit : 134217728 soft_limit : unlimited swap_hard_limit: unlimited # cat /sys/fs/cgroup/memory/machine.slice/machine-qemu\\x2d6\\x2duefi1.scope/libvirt/memory.limit_in_bytes 137438953472 When vm starts with memtune->hard_limit set in domain XML, memlock hard limit( hard_limit of locked memory, shown by 'prlimit -l')will be set to the value of memtune->hard_limit. This's probably because memlock hard limit must be less than memtune->hard_limit.
Re: Memory locking limit and zero-copy migrations
Fangge Jin writes: > On Thu, Aug 18, 2022 at 2:46 PM Milan Zamazal wrote: > >> Fangge Jin writes: >> >> > I can share some test results with you: >> > 1. If no memtune->hard_limit is set when start a vm, the default memlock >> > hard limit is 64MB >> > 2. If memtune->hard_limit is set when start a vm, memlock hard limit will >> > be set to the value of memtune->hard_limit >> > 3. If memtune->hard_limit is updated at run-time, memlock hard limit >> won't >> > be changed accordingly >> > >> > And some additional knowledge: >> > 1. memlock hard limit can be shown by ‘prlimit -p -l’ >> > 2. The default value of memlock hard limit can be changed by setting >> > LimitMEMLOCK in /usr/lib/systemd/system/virtqemud.service >> >> Ah, that explains it to me, thank you. And since in the default case >> the systemd limit is not reported in of a running VM, I assume >> libvirt takes it as "not set" and sets the higher limit when setting up >> a zero-copy migration. Good. >> > Not sure whether you already know this, but I had a hard time > differentiating the two concepts: > 1. memlock hard limit(shown by prlimit): the hard limit for locked host > memory > 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use host > memory, this memory can be swapped out. No, I didn't know it, thank you for pointing this out. Indeed, 2. is what both the libvirt and kernel documentation seem to say, although not so clearly. But when I add with to the domain XML and then start the VM, I can see the limit shown by `prlimit -l' is increased accordingly. This is good for my use case, but does it match what you say about the two concepts?
Re: Memory locking limit and zero-copy migrations
On Thu, Aug 18, 2022 at 2:46 PM Milan Zamazal wrote: > Fangge Jin writes: > > > I can share some test results with you: > > 1. If no memtune->hard_limit is set when start a vm, the default memlock > > hard limit is 64MB > > 2. If memtune->hard_limit is set when start a vm, memlock hard limit will > > be set to the value of memtune->hard_limit > > 3. If memtune->hard_limit is updated at run-time, memlock hard limit > won't > > be changed accordingly > > > > And some additional knowledge: > > 1. memlock hard limit can be shown by ‘prlimit -p -l’ > > 2. The default value of memlock hard limit can be changed by setting > > LimitMEMLOCK in /usr/lib/systemd/system/virtqemud.service > > Ah, that explains it to me, thank you. And since in the default case > the systemd limit is not reported in of a running VM, I assume > libvirt takes it as "not set" and sets the higher limit when setting up > a zero-copy migration. Good. > Not sure whether you already know this, but I had a hard time differentiating the two concepts: 1. memlock hard limit(shown by prlimit): the hard limit for locked host memory 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use host memory, this memory can be swapped out. > > Regards, > Milan >
Re: Memory locking limit and zero-copy migrations
Fangge Jin writes: > I can share some test results with you: > 1. If no memtune->hard_limit is set when start a vm, the default memlock > hard limit is 64MB > 2. If memtune->hard_limit is set when start a vm, memlock hard limit will > be set to the value of memtune->hard_limit > 3. If memtune->hard_limit is updated at run-time, memlock hard limit won't > be changed accordingly > > And some additional knowledge: > 1. memlock hard limit can be shown by ‘prlimit -p -l’ > 2. The default value of memlock hard limit can be changed by setting > LimitMEMLOCK in /usr/lib/systemd/system/virtqemud.service Ah, that explains it to me, thank you. And since in the default case the systemd limit is not reported in of a running VM, I assume libvirt takes it as "not set" and sets the higher limit when setting up a zero-copy migration. Good. Regards, Milan > BR, > Fangge Jin > > On Wed, Aug 17, 2022 at 19:25 Milan Zamazal wrote: > >> Peter Krempa writes: >> >> > On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal wrote: >> >> Hi, >> >> >> > >> >> do I read libvirt sources right that when is not used in the >> >> libvirt domain then libvirt takes proper care about setting memory >> >> locking limits when zero-copy is requested for a migration? >> > >> > Well yes, for a definition of "proper". In this instance qemu can lock >> > up to the guest-visible memory size of memory for the migration, thus we >> > set the lockable size to the guest memory size. This is a simple upper >> > bound which is supposed to work in all scenarios. Qemu is also unlikely >> > to ever use up all the allowed locking. >> >> Great, thank you for confirmation. >> >> >> I also wonder whether there are any other situations where memory limits >> >> could be set by libvirt or QEMU automatically rather than having no >> >> memory limits? We had oVirt bugs in the past where certain VMs with >> >> VFIO devices couldn't be started due to extra requirements on the amount >> >> of locked memory and adding to the domain apparently >> >> helped. >> > >> > is not only an amount of memory qemu can lock into ram, but >> > an upper bound of all memory the qemu process can consume. This includes >> > any qemu overhead e.g. used for the emulation layer. >> > >> > Guessing the correct size of overhead still has the same problems it had >> > and libvirt is not going to be in the business of doing that. >> >> To clarify, my point was not whether libvirt should, but whether libvirt >> or any related component possibly does (or did in the past) impose >> memory limits. Because as I was looking around it seems there are no >> real memory limits by default, at least in libvirt, but some limit had >> been apparently hit in the reported bugs. >> >>
Re: Memory locking limit and zero-copy migrations
On Thu, Aug 18, 2022 at 10:46 AM Fangge Jin wrote: > I can share some test results with you: > 1. If no memtune->hard_limit is set when start a vm, the default memlock > hard limit is 64MB > 2. If memtune->hard_limit is set when start a vm, memlock hard limit will > be set to the value of memtune->hard_limit > 3. If memtune->hard_limit is updated at run-time, memlock hard limit won't > be changed accordingly > > And some additional knowledge: > 1. memlock hard limit can be shown by ‘prlimit -p -l’ > 2. The default value of memlock hard limit can be changed by setting > LimitMEMLOCK in /usr/lib/systemd/system/virtqemud.service > Or /usr/lib/systemd/system/libvirtd.service here. > > BR, > Fangge Jin >
Re: Memory locking limit and zero-copy migrations
I can share some test results with you: 1. If no memtune->hard_limit is set when start a vm, the default memlock hard limit is 64MB 2. If memtune->hard_limit is set when start a vm, memlock hard limit will be set to the value of memtune->hard_limit 3. If memtune->hard_limit is updated at run-time, memlock hard limit won't be changed accordingly And some additional knowledge: 1. memlock hard limit can be shown by ‘prlimit -p -l’ 2. The default value of memlock hard limit can be changed by setting LimitMEMLOCK in /usr/lib/systemd/system/virtqemud.service BR, Fangge Jin On Wed, Aug 17, 2022 at 19:25 Milan Zamazal wrote: > Peter Krempa writes: > > > On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal wrote: > >> Hi, > >> > > > >> do I read libvirt sources right that when is not used in the > >> libvirt domain then libvirt takes proper care about setting memory > >> locking limits when zero-copy is requested for a migration? > > > > Well yes, for a definition of "proper". In this instance qemu can lock > > up to the guest-visible memory size of memory for the migration, thus we > > set the lockable size to the guest memory size. This is a simple upper > > bound which is supposed to work in all scenarios. Qemu is also unlikely > > to ever use up all the allowed locking. > > Great, thank you for confirmation. > > >> I also wonder whether there are any other situations where memory limits > >> could be set by libvirt or QEMU automatically rather than having no > >> memory limits? We had oVirt bugs in the past where certain VMs with > >> VFIO devices couldn't be started due to extra requirements on the amount > >> of locked memory and adding to the domain apparently > >> helped. > > > > is not only an amount of memory qemu can lock into ram, but > > an upper bound of all memory the qemu process can consume. This includes > > any qemu overhead e.g. used for the emulation layer. > > > > Guessing the correct size of overhead still has the same problems it had > > and libvirt is not going to be in the business of doing that. > > To clarify, my point was not whether libvirt should, but whether libvirt > or any related component possibly does (or did in the past) impose > memory limits. Because as I was looking around it seems there are no > real memory limits by default, at least in libvirt, but some limit had > been apparently hit in the reported bugs. > >
Re: Memory locking limit and zero-copy migrations
Peter Krempa writes: > On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal wrote: >> Hi, >> > >> do I read libvirt sources right that when is not used in the >> libvirt domain then libvirt takes proper care about setting memory >> locking limits when zero-copy is requested for a migration? > > Well yes, for a definition of "proper". In this instance qemu can lock > up to the guest-visible memory size of memory for the migration, thus we > set the lockable size to the guest memory size. This is a simple upper > bound which is supposed to work in all scenarios. Qemu is also unlikely > to ever use up all the allowed locking. Great, thank you for confirmation. >> I also wonder whether there are any other situations where memory limits >> could be set by libvirt or QEMU automatically rather than having no >> memory limits? We had oVirt bugs in the past where certain VMs with >> VFIO devices couldn't be started due to extra requirements on the amount >> of locked memory and adding to the domain apparently >> helped. > > is not only an amount of memory qemu can lock into ram, but > an upper bound of all memory the qemu process can consume. This includes > any qemu overhead e.g. used for the emulation layer. > > Guessing the correct size of overhead still has the same problems it had > and libvirt is not going to be in the business of doing that. To clarify, my point was not whether libvirt should, but whether libvirt or any related component possibly does (or did in the past) impose memory limits. Because as I was looking around it seems there are no real memory limits by default, at least in libvirt, but some limit had been apparently hit in the reported bugs.
Re: Memory locking limit and zero-copy migrations
On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal wrote: > Hi, > > do I read libvirt sources right that when is not used in the > libvirt domain then libvirt takes proper care about setting memory > locking limits when zero-copy is requested for a migration? Well yes, for a definition of "proper". In this instance qemu can lock up to the guest-visible memory size of memory for the migration, thus we set the lockable size to the guest memory size. This is a simple upper bound which is supposed to work in all scenarios. Qemu is also unlikely to ever use up all the allowed locking. > I also wonder whether there are any other situations where memory limits > could be set by libvirt or QEMU automatically rather than having no > memory limits? We had oVirt bugs in the past where certain VMs with > VFIO devices couldn't be started due to extra requirements on the amount > of locked memory and adding to the domain apparently > helped. is not only an amount of memory qemu can lock into ram, but an upper bound of all memory the qemu process can consume. This includes any qemu overhead e.g. used for the emulation layer. Guessing the correct size of overhead still has the same problems it had and libvirt is not going to be in the business of doing that.
Memory locking limit and zero-copy migrations
Hi, do I read libvirt sources right that when is not used in the libvirt domain then libvirt takes proper care about setting memory locking limits when zero-copy is requested for a migration? I also wonder whether there are any other situations where memory limits could be set by libvirt or QEMU automatically rather than having no memory limits? We had oVirt bugs in the past where certain VMs with VFIO devices couldn't be started due to extra requirements on the amount of locked memory and adding to the domain apparently helped. Thanks, Milan
