On 6/22/26 16:29, David Hildenbrand (Arm) wrote:
> This email originated from an IP that might not be authorized by the domain
> it was sent from.
> Do not click links or open attachments unless it is an email you expected to
> receive.
> On 6/22/26 15:37, Denis V. Lunev wrote:
>> Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on
>> device_shutdown()") the virtio bus breaks and resets every virtio device
>> during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no
>> .shutdown of its own, so that generic path runs while the balloon's
>> asynchronous work is still armed: the free page reporting worker, the
>> inflate/deflate and stats workers, the OOM notifier and the free page
>> shrinker.
>>
>> Once the device has been broken, virtqueue_add_inbuf() in
>> virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE().
>> On a kernel booted with panic_on_warn that turns an ordinary reboot into a
>> fatal panic in the middle of device_shutdown(), so the machine never
>> reaches the new kernel. The inflate/deflate and OOM paths do not warn but
>> are no better off: they call wait_event(vb->acked, ...) and would block
>> forever on a queue that can no longer complete.
>>
>> This was hit in the field as an intermittent failure of a virtualization
>> cluster upgrade: guest storage nodes were rebooted via kexec into the new
>> kernel, and the ones whose free page reporting happened to run during
>> device_shutdown() panicked (the guests run with panic_on_warn) and never
>> came back, stalling the rolling upgrade. The crash dump showed the WARN at
>> virtio_balloon.c:216 in a page_reporting kworker, with all the balloon
>> virtqueues already broken.
>>
>> Patch 1 factors the teardown out of virtballoon_remove() into a
>> virtballoon_quiesce() helper (no functional change). Patch 2 adds a
>> virtio_balloon .shutdown handler that quiesces via that helper while the
>> device is still alive, then breaks and resets it the way the generic
>> virtio_dev_shutdown() would.
>>
>> Relaxing the single WARN_ON_ONCE() instead was considered and rejected: it
>> would silence the panic but leave the inflate/deflate and OOM paths
>> hanging on the broken device. The device has to be quiesced, not just kept
>> quiet.
> Do you have a link to that discussion you could add?
>
>
that was internal discussion for that fix. We have faced this
during tests, made the fix and validated it before sending.
Thus - sorry, this is my mistake.
Den
