On 9/1/26 11:35, Daniel P. Berrangé wrote: > On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote: >> On 9/1/26 00:53, Jim Fehlig wrote: >>> On 8/31/26 9:34 AM, Peter Krempa wrote: >>>> On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote: >>>>> From: Jim Fehlig <[email protected]> >>>>> >>>>> When shutting down a VM, libvirt sends the associated QEMU process >>>>> SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, >>>>> then waits up to another 30 seconds for the process to exit before >>>>> reporting and returning an error. Commit be2ca04447 added 2 seconds per >>>>> assigned host device to the total time libvirt waits for a QEMU process >>>>> to terminate. Other scenarios than the one described in be2ca04447 could >>>>> delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large >>>>> memory allocations backed by 4k pages on the host. >>>>> >>>>> Instead of trying to cover all such scenarios based on VM configuration, >>>>> introduce a 'process_exit_wait' setting in qemu.conf to control how much >>>>> additional time (in seconds) libvirt will wait for a QEMU process to >>>>> terminate before reporting an error. >>>> >>>> You are stating that an error is reported. Can you please elaborate when >>>> you are seeing such an error? >>> >>> When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the >>> host >> >> We recommend to use hugepages to back large VMs, but for flexibility reasons >> and other >> constraints, users end up with these mid to large size VMs backed by normal >> memory. >> >> We have reports of this occuring on NUMA servers with even smaller VMs, f.e >> 512GiB. >> >> Normally with a fairly modern enterprise CPU and RAM this can take ~25s to >> terminate the QEMU process in isolation, but in certain conditions this can >> increase >> up to the 40 seconds. >> >> This was fine up until some time ago since the timeout was bugged and did >> not trigger: as the timeout was fixed, users started seeing their use cases >> break. > > FWIW, IBM developed a feature in QEMU/libvirt to deal with this > > <async-teardown enabled='yes'/> > > an implication of this approach though is that the RAM is not immediately > available for launching a new guest, and of course it requires per-guest > config knobs to be enabled. > > >>>>> diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in >>>>> index 97b0141cf6..2f24059053 100644 >>>>> --- a/src/qemu/qemu.conf.in >>>>> +++ b/src/qemu/qemu.conf.in >>>>> @@ -844,6 +844,18 @@ >>>>> #max_threads_per_process = 0 >>>>> >>>>> >>>>> +# When shutting down a VM, libvirt will wait up to 40 seconds for >>>>> +# the associated QEMU process to exit before reporting an error. >>>>> +# For some VM configurations, QEMU might require more time to >>>>> +# cleanup and exit, e.g. VMs with very large memory allocations. >>>>> +# >>>>> +# If process_exit_wait is set to a positive interger, libvirt >>>>> +# will use the value as additional time to wait for the QEMU >>>>> +# process to exit before reporting it cannot be terminated. >>>>> +# >>>>> +#process_exit_wait = 0 >>>> >>>> I don't like this as a global option. The specifics of a VM can be >>>> vastly different and setting this globally will possibly be unable to >>>> satisfy new VMs without restart of the daemon. >> >> FWIW, in my view it would be both more convenient and more meaningful >> for the admin to configure this as a single global parameter when planning >> the host configuration, resources and workloads. >> >> A restart of the daemon if the admin doesn't get it "right" the first time >> is not that bad imo, existing VMs continue to run, and libvirt daemons >> restart fairly quickly. >> >> The thing to keep in mind for an admin / provisioning rule / agent /... is >> the max size of a normal memory backed VM that could land on this host, >> and the worst case pressure the workloads on the host / socket can generate >> on the memory system, given the host architecture and topology. > > The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was > designed from the POV that a normally behaving QEMU will "die" very > quickly. IOW, any scenario where we reached the KILL stage was almost > certainly a broken QEMU/kernel in some respect. > > Clearly this is no longer a valid assumption. When "normal" behaviour > or QEMU no longer matches libvirt's default mgmt action behaviour > then I don't think a global qemu.conf setting or a per-VM setting is > the ideal approach. > > We need to ensure libvirt "does the right thing" out of the box, as > best as we can.
Hi Daniel, in principle I agree it would be the best outcome, just seems hard to get it right. > > IMHO, this suggests we need to dynamically increase our wait time > before KILL based on the guest RAM size. eg Add 5 seconds for each > 100 GB of small page RAM. I pulled that number out of the air, > you would need to pick something better based on a typical system, > plus some buffer/fuzz. > > > Also I've noticed that TDX guests are painfully slow to teardown, > even with tiny RAM sizes. So we might need to increase wait times > even more when using TDX. Indeed. And like this there are many more factors that can change the result substantially (hugepages settings, NUMA, cpu and load state @memory controller, ...). Each one of these factors would need to be extracted or sampled and multiplied for the total RAM size. In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket. I am concerned that getting to the "right" number might prove impractical? > > > With regards, > Daniel Thanks! CLaudio
