On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote: > On 9/1/26 00:53, Jim Fehlig wrote: > > On 8/31/26 9:34 AM, Peter Krempa wrote: > >> On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote: > >>> From: Jim Fehlig <[email protected]> > >>> > >>> When shutting down a VM, libvirt sends the associated QEMU process > >>> SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, > >>> then waits up to another 30 seconds for the process to exit before > >>> reporting and returning an error. Commit be2ca04447 added 2 seconds per > >>> assigned host device to the total time libvirt waits for a QEMU process > >>> to terminate. Other scenarios than the one described in be2ca04447 could > >>> delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large > >>> memory allocations backed by 4k pages on the host. > >>> > >>> Instead of trying to cover all such scenarios based on VM configuration, > >>> introduce a 'process_exit_wait' setting in qemu.conf to control how much > >>> additional time (in seconds) libvirt will wait for a QEMU process to > >>> terminate before reporting an error. > >> > >> You are stating that an error is reported. Can you please elaborate when > >> you are seeing such an error? > > > > When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the > > host > > We recommend to use hugepages to back large VMs, but for flexibility reasons > and other > constraints, users end up with these mid to large size VMs backed by normal > memory. > > We have reports of this occuring on NUMA servers with even smaller VMs, f.e > 512GiB. > > Normally with a fairly modern enterprise CPU and RAM this can take ~25s to > terminate the QEMU process in isolation, but in certain conditions this can > increase > up to the 40 seconds. > > This was fine up until some time ago since the timeout was bugged and did > not trigger: as the timeout was fixed, users started seeing their use cases > break.
FWIW, IBM developed a feature in QEMU/libvirt to deal with this <async-teardown enabled='yes'/> an implication of this approach though is that the RAM is not immediately available for launching a new guest, and of course it requires per-guest config knobs to be enabled. > >>> diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in > >>> index 97b0141cf6..2f24059053 100644 > >>> --- a/src/qemu/qemu.conf.in > >>> +++ b/src/qemu/qemu.conf.in > >>> @@ -844,6 +844,18 @@ > >>> #max_threads_per_process = 0 > >>> > >>> > >>> +# When shutting down a VM, libvirt will wait up to 40 seconds for > >>> +# the associated QEMU process to exit before reporting an error. > >>> +# For some VM configurations, QEMU might require more time to > >>> +# cleanup and exit, e.g. VMs with very large memory allocations. > >>> +# > >>> +# If process_exit_wait is set to a positive interger, libvirt > >>> +# will use the value as additional time to wait for the QEMU > >>> +# process to exit before reporting it cannot be terminated. > >>> +# > >>> +#process_exit_wait = 0 > >> > >> I don't like this as a global option. The specifics of a VM can be > >> vastly different and setting this globally will possibly be unable to > >> satisfy new VMs without restart of the daemon. > > FWIW, in my view it would be both more convenient and more meaningful > for the admin to configure this as a single global parameter when planning > the host configuration, resources and workloads. > > A restart of the daemon if the admin doesn't get it "right" the first time > is not that bad imo, existing VMs continue to run, and libvirt daemons > restart fairly quickly. > > The thing to keep in mind for an admin / provisioning rule / agent /... is > the max size of a normal memory backed VM that could land on this host, > and the worst case pressure the workloads on the host / socket can generate > on the memory system, given the host architecture and topology. The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect. Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach. We need to ensure libvirt "does the right thing" out of the box, as best as we can. IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz. Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
