[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-07-07 Thread Launchpad Bug Tracker
[Expired for QEMU because there has been no activity for 60 days.] ** Changed in: qemu Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1894804 Title: Second

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-07-07 Thread Launchpad Bug Tracker
[Expired for qemu (Ubuntu) because there has been no activity for 60 days.] ** Changed in: qemu (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1894804

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-05-08 Thread Thomas Huth
Is there still anything left to do here for upstream QEMU? ** Changed in: qemu Status: New => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1894804 Title: Second DEVICE_DELETED

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-03 Thread Christian Ehrhardt 
I was seeing in my manual tests that the delete->Event was taking a bit longer, maybe your retry logic kicks in before it is complete now. And due to that the problem takes place. So with the PPA you should get: a) a warning, but no more the cancelled/undefined behavior on double delete b) can

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-02 Thread Lee Yarwood
Thanks Christian, I'll confirm that we see error from [2] with that PPA shortly. In the meantime I've removed the device detach retry logic from OpenStack Nova and now always see two DEVICE_DELETED events raised by QEMU after a single device_del request from libvirt:

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-02 Thread Christian Ehrhardt 
Thanks Lee, builds are most reliably done in Lauchpad IMHO and the Ubuntu Delta is a quilt stack which isn't as easily bisectable. If we end up searching not between Ubuntu Delta I can help to get you qemu built from git for that. But for some initially probing which range of changes we want to

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood
Here's an example from QMP_requests_and_replies_from_libvirtd.log.gz attached earlier to the bug by Kashyap. We can see two device_del commands prior to only seeing a single event emitted by QEMU: http://paste.openstack.org/show/798607/ # zgrep 0x7f768806eaf0

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood
While testing >=v5.0.0 on Fedora 32 I've stumbled across the following change in QEMU: https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b The change suggests that previously parallel requests to detach a PCIe device (such as virtio-blk) from QEMU would result in the

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood
> @Christian Yes we can test this in OpenStack CI with PPAs if you > could provide Bionic based builds. I'm not sure what I was thinking here but we *can* use Focal based builds of QEMU in CI now. -- You received this bug notification because you are a member of qemu- devel-ml, which is

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-30 Thread Lee Yarwood
@Christian Yes we can test this in OpenStack CI with PPAs if you could provide Bionic based builds. Better yet I could even try to setup a git bisect locally if you have steps to build from a source tree somewhere? @James Did you try using OpenStack CI sized hosts (8 vCPUs, 8GB of RAM) and the

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-29 Thread Christian Ehrhardt 
Thanks James, but now I'm unsure where to go from here as it isn't reproducible with many tries at different scales that James and I did. @Sean/Lee Since you wondered if it might be due to Ubuntu Delta on top of 4.2 - there are two things we could compare Ubuntu's qemu to then: 1. qemu 4.2 as

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-25 Thread James Page
I'm not able to reproduce at this point in time - as the version of libvirt/qemu is the same I deployed an Ussuri cloud on focal and ran block device add/remove with concurrency for several hundred iterations but I was not able to trigger a detach failure. -- You received this bug notification

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-24 Thread Christian Ehrhardt 
As outlined before the time from device_del to the DEVICE_DELETED is indeed "a bit longer" in Focal (from ~0.1s to ~6s), but other than that I couldn't find anything else yet. We were wondering due to [1] in a related bug if it might be dependent to (over)load. I was consuming Disk and CPU in a

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-21 Thread Christian Ehrhardt 
Hi, interesting bug report - thanks lee, Kashyap and Sean - as well as Danil for taking a look already. If this would always fail no unplugs would work ever which I knew can't be right as I test that. So we need to find what is different... @Openstack people - is that reliably triggering at

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-16 Thread sean mooney
hum i was hoping to indicate this affect focal in some what but not sure how to do that but this issue happens with the ubunutu 20.04 version of qemu 4.2 it does not seam to happen with the centos 8 build of the same qemu so i dont know if there is a delta in packages or if its just a case that

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Daniel Berrange
Note that there are many VMs running, so it is important to isolate logs for just one of them, by matching on mon=0x. For example considering this $ grep 0x7f021c0b9c70 libvirtd.log | grep -E 'QEMU_MONITOR_(RECV_REPLY|RECV_EVENT|SEND_MSG)' We can see the "device_del" request:

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Kashyap Chamarthy
Attaching the QMP requests and responses filtered out from the giant libvirtd.log. This file was generated using: $> grep -Ei '(QEMU_MONITOR_SEND_MSG|QEMU_MONITOR_RECV_)' libvirtd.log |& tee QMP_requests_and_replies_from_libvirtd.log (And then I `gzip`ed it; compressed - 3.1MB;

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Kashyap Chamarthy
Just to expand a little more on the "second DEVICE_DELETED" event: Apparently this is an "oddity" of VirtIO devices where we see _two_ DEVICE_DELETED events upon hot-unplugging a device. So, ideally, we should see the following two events, but we're only seeing the first one: (1) "event":