** Description changed:
+
+ [Impact]
+
+ The libvirt RPC client (src/rpc/virnetclient.c) leaks a GSource object every
+ time a thread has to interrupt the main event loop to force another thread out
+ of poll(). The reference on the idle GSource created for the wakeup is never
+ released, and the source is never removed from the main context when the
thread
+ is woken up before the wakeup callback fires.
+
+ Over time this produces an ever-growing list of GSource objects attached to
the
+ main context, the libvirtd daemon becomes progressively slower.
+
+ Real-world impact reported by multiple users on 24.04:
+ - OpenStack nova-compute eventually pins a CPU at 100% because periodic tasks
+ that call virConnectListAllNodeDevices + virNodeDeviceNumOfCaps +
+ virNodeDeviceListCaps get slower and slower.
+ - The Apache CloudStack agent is affected the same way (LP #2133531,
+ CloudStack discussion #12450).
+ - Reporters observe CPU usage dropping from ~6% to ~0.5% after applying the
+ fix.
+
+ This is fixed upstream since libvirt v10.5.0 by commit
+ 98f1cf88fa7e0f992d93f376418fbfb3996a9690, but has not been backported to the
+ 10.0.0-2ubuntu8.x series in 24.04 (Noble).
+
+ [Test Plan]
+
+ On an Ubuntu 24.04 host with libvirtd running (ideally with some active node
+ devices), run the following Python script against the unpatched package:
+
+ import libvirt
+ import time
+ import threading
+
+ conn = libvirt.open("qemu:///system")
+
+ def run():
+ while True:
+ t0 = time.time()
+ try:
+ for dev in conn.listAllDevices(0):
+ dev.listCaps()
+ except Exception:
+ print("Exception")
+ print(time.time() - t0)
+
+ t1 = threading.Thread(target=run)
+ t1.start()
+ t2 = threading.Thread(target=run)
+ t2.start()
+
+ The script runs two threads that repeatedly list all node devices and their
+ capabilities, printing the time per iteration.
+
+ - Unpatched: the per-iteration time grows steadily (the client gets slower
and
+ slower), and libvirtd CPU usage climbs over time.
+ - Patched: the per-iteration time stays stable and does not degrade, and CPU
+ usage stays low.
+
+ A longer-running confirmation (leaving libvirtd up for many hours) should show
+ CPU usage stabilizing at a low level rather than continually increasing.
+
+ [Where problems could occur]
+
+ - The change is confined to virNetClientIO() in src/rpc/virnetclient.c. It
+ switches the wakeup GSource to g_autoptr(GSource) and adds an explicit
+ g_source_destroy(wakeup) after the thread is woken. Any regression would
+ surface in the RPC client's multi-threaded wakeup/dispatch path.
+ - A mistake in the source lifetime handling could theoretically cause a
+ use-after-free or a missed wakeup, which would manifest as RPC client hangs
+ or crashes under concurrent access. This risk is low: the fix is a small,
+ well-reviewed upstream change that has been
+ in libvirt since v10.5.0 with no follow-up regressions.
+ - The affected code path is exercised whenever multiple threads use the same
+ RPC connection concurrently, so any regression would be caught quickly by
the
+ test plan above and by the OpenStack/CloudStack workloads that reported the
+ original issue.
+
+ [Other Info]
+
+ - Upstream fix:
+
https://gitlab.com/libvirt/libvirt/-/commit/98f1cf88fa7e0f992d93f376418fbfb3996a9690
+ - Fixed upstream in libvirt v10.5.0; so ONLY affects Noble or older.
+
+ ---Original bug report---
+
Ubuntu release: 24.04.4
libvirt version: 10.0.0-2ubuntu8.12
We noticed that the libvirt RPC client becomes slower and slower over
time. In our setup it's used from an OpenStack nova-compute service. At
some point the nova-compute service is constantly taking 100% CPU
because a periodic task that lists all node devices and their caps takes
all the time.
The periodic task calls virConnectListAllNodeDevices followed by calls
per device of virNodeDeviceNumOfCaps and virNodeDeviceListCaps. And
those RPCs become slower and slower.
This is probably related to a GSource leak in src/rpc/virnetclient.c
which is already fixed by libvirt v10.5.0. See commit
https://gitlab.com/libvirt/libvirt/-/commit/98f1cf88fa7e0f992d93f376418fbfb3996a9690
The commit message says:
(quote)
We never release the reference on the GSource created for
interrupting the main loop, nor do we remove it from the
main context if our thread is woken up prior to the wakeup
callback firing.
This can result in a leak of GSource objects, along with an
ever growing list of GSources attached to the main context,
which will gradually slow down execution of the loop, as
several operations are O(N) for the number of attached GSource
objects.
(end quote)
This patch has not been backported yet to 10.0.0-2ubuntu8.x for Ubuntu
24.04
** Changed in: libvirt (Ubuntu Noble)
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2142757
Title:
libvirt 10.0.0-2ubuntu8.x - rpc: leak of GSource in use for
interrupting main loop
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2142757/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
