Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on
device_shutdown()") the virtio bus breaks and resets every virtio device
during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no
.shutdown of its own, so that generic path runs while the balloon's
asynchronous work is still armed: the free page reporting worker, the
inflate/deflate and stats workers, the OOM notifier and the free page
shrinker.Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete. This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken. Patch 1 factors the teardown out of virtballoon_remove() into a virtballoon_quiesce() helper (no functional change). Patch 2 adds a virtio_balloon .shutdown handler that quiesces via that helper while the device is still alive, then breaks and resets it the way the generic virtio_dev_shutdown() would. Relaxing the single WARN_ON_ONCE() instead was considered and rejected: it would silence the panic but leave the inflate/deflate and OOM paths hanging on the broken device. The device has to be quiesced, not just kept quiet. Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched module reproduces the WARN within a couple of cycles, while the patched module survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean on both patches. Denis V. Lunev (2): virtio_balloon: factor out virtballoon_quiesce() virtio_balloon: quiesce balloon work before device shutdown drivers/virtio/virtio_balloon.c | 37 ++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) -- 2.53.0
