subject:"\[Bug 1894804\] Re\: Second DEVICE_DELETED event missing during virtio\-blk disk device detach"

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-07-07 Thread Launchpad Bug Tracker

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  Expired
Status in qemu package in Ubuntu:
  Expired

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.config \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \
  -name guest=instance-003a,debug-threads=on \
  -S \
  -object

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-07-07 Thread Launchpad Bug Tracker

[Expired for qemu (Ubuntu) because there has been no activity for 60
days.]

** Changed in: qemu (Ubuntu)
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  Expired
Status in qemu package in Ubuntu:
  Expired

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.config \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \
  -name guest=instance-003a,debug-threads=on \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2021-05-08 Thread Thomas Huth

Is there still anything left to do here for upstream QEMU?

** Changed in: qemu
   Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  Incomplete
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.config \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \
  -name guest=instance-003a,debug-threads=on \
  -S \
  -object

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-03 Thread Christian Ehrhardt 

I was seeing in my manual tests that the delete->Event was taking a bit
longer, maybe your retry logic kicks in before it is complete now. And
due to that the problem takes place.

So with the PPA you should get:
a) a warning, but no more the cancelled/undefined behavior on double delete
b) can tune your retry logic to no more trigger the warning messages

So I wasn't sure from the logs - is the fix in the PPA good to overcome
the issue completely or is it missing something?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-02 Thread Lee Yarwood

Thanks Christian,

I'll confirm that we see error from [2] with that PPA shortly.

In the meantime I've removed the device detach retry logic from
OpenStack Nova and now always see two DEVICE_DELETED events raised by
QEMU after a single device_del request from libvirt:

http://paste.openstack.org/show/798637/

# zgrep 'debug.*"event": "DEVICE_DELETED"' libvirtd_log.txt.gz 
2020-10-01 21:37:33.180+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588253, "microseconds": 179881}, "event": 
"DEVICE_DELETED", "data": {"path": 
"/machine/peripheral/virtio-disk1/virtio-backend"}}]
2020-10-01 21:37:33.242+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588253, "microseconds": 242565}, "event": 
"DEVICE_DELETED", "data": {"device": "virtio-disk1", "path": 
"/machine/peripheral/virtio-disk1"}}]

2020-10-01 21:40:17.062+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588417, "microseconds": 62704}, "event": 
"DEVICE_DELETED", "data": {"path": 
"/machine/peripheral/virtio-disk1/virtio-backend"}}]
2020-10-01 21:40:17.114+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588417, "microseconds": 113888}, "event": 
"DEVICE_DELETED", "data": {"device": "virtio-disk1", "path": 
"/machine/peripheral/virtio-disk1"}}]

2020-10-01 21:40:20.911+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588420, "microseconds": 911560}, "event": 
"DEVICE_DELETED", "data": {"path": 
"/machine/peripheral/virtio-disk1/virtio-backend"}}]
2020-10-01 21:40:20.985+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588420, "microseconds": 985753}, "event": 
"DEVICE_DELETED", "data": {"device": "virtio-disk1", "path": 
"/machine/peripheral/virtio-disk1"}}]

2020-10-01 21:42:53.528+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588573, "microseconds": 528330}, "event": 
"DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}]
2020-10-01 21:42:53.583+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588573, "microseconds": 583063}, "event": 
"DEVICE_DELETED", "data": {"device": "net1", "path": 
"/machine/peripheral/net1"}}]

2020-10-01 21:44:21.156+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588661, "microseconds": 155903}, "event": 
"DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}]
2020-10-01 21:44:21.214+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588661, "microseconds": 213714}, "event": 
"DEVICE_DELETED", "data": {"device": "net1", "path": 
"/machine/peripheral/net1"}}]

2020-10-01 21:45:55.422+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588755, "microseconds": 422238}, "event": 
"DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}]
2020-10-01 21:45:55.479+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588755, "microseconds": 479250}, "event": 
"DEVICE_DELETED", "data": {"device": "net1", "path": 
"/machine/peripheral/net1"}}]

2020-10-01 21:46:09.001+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588769, "microseconds": 970}, "event": 
"DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}]
2020-10-01 21:46:09.059+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601588769, "microseconds": 58799}, "event": 
"DEVICE_DELETED", "data": {"device": "net1", "path": 
"/machine/peripheral/net1"}}]

2020-10-01 21:48:05.182+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 160155, "microseconds": 182382}, "event": 
"DEVICE_DELETED", "data": {"path": 
"/machine/peripheral/virtio-disk1/virtio-backend"}}]
2020-10-01 21:48:05.246+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 160155, "microseconds": 246077}, "event": 
"DEVICE_DELETED", "data": {"device": "virtio-disk1", "path": 
"/machine/peripheral/virtio-disk1"}}]

2020-10-01 21:48:08.211+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 160158, "microseconds": 211386}, "event": 
"DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}]
2020-10-01 21:48:08.269+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 160158, "microseconds": 269371}, "event": 
"DEVICE_DELETED", "data": {"device": "net1", "path": 
"/machine/peripheral/net1"}}]

2020-10-01 21:51:58.384+: 55842: debug : qemuMonitorJSONIOProcessLine:220 : 
Line [{"timestamp": {"seconds": 1601589118, "microseconds": 384455}, "event":

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-02 Thread Christian Ehrhardt 

Thanks Lee,
builds are most reliably done in Lauchpad IMHO and the Ubuntu Delta is a quilt 
stack which isn't as easily bisectable.
If we end up searching not between Ubuntu Delta I can help to get you qemu 
built from git for that.
But for some initially probing which range of changes we want to actually look 
at let me provide you PPAs.

Try #1 in PPA [1] is the fix you referred [2] to that avoids the double
delete to cause unexpected results. It will also add the warnings you
have seen. So giving it a test should be a great first try.

Let me know what the results with that are.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4294/+packages
[2]: 
https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood

Here's an example from QMP_requests_and_replies_from_libvirtd.log.gz
attached earlier to the bug by Kashyap. We can see two device_del
commands prior to only seeing a single event emitted by QEMU:

http://paste.openstack.org/show/798607/

# zgrep 0x7f768806eaf0 QMP_requests_and_replies_from_libvirtd.log.gz | egrep 
'(device_del|DEVICE_DELETED)'
2020-09-07 19:50:11.315+: 65558: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-367"}
2020-09-07 19:50:17.535+: 65557: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-369"}
2020-09-07 19:50:18.138+: 6: info : qemuMonitorJSONIOProcessLine:234 : 
QEMU_MONITOR_RECV_EVENT: mon=0x7f768806eaf0 event={"timestamp": {"seconds": 
1599508218, "microseconds": 138046}, "event": "DEVICE_DELETED", "data": 
{"path": "/machine/peripheral/virtio-disk1/virtio-backend"}}
2020-09-07 19:50:24.549+: 65556: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-371"}
2020-09-07 19:50:33.561+: 65556: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-373"}
2020-09-07 19:50:44.572+: 65557: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-375"}
2020-09-07 19:50:57.583+: 65558: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-377"}
2020-09-07 19:51:12.595+: 65556: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-379"}
2020-09-07 19:51:30.548+: 65557: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-381"}
2020-09-07 19:51:51.804+: 65559: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-383"}
2020-09-07 19:53:33.286+: 65559: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-385"}
2020-09-07 19:53:40.310+: 65558: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-387"}
2020-09-07 19:53:49.353+: 65556: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-389"}
2020-09-07 19:54:00.771+: 65560: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-391"}
2020-09-07 19:54:15.737+: 65559: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-393"}
2020-09-07 19:54:30.767+: 65557: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-395"}
2020-09-07 19:54:47.778+: 65560: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-397"}
2020-09-07 19:55:06.793+: 65559: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f768806eaf0 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-399"}

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood

While testing >=v5.0.0 on Fedora 32 I've stumbled across the following
change in QEMU:

https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b

The change suggests that previously parallel requests to detach a PCIe
device (such as virtio-blk) from QEMU would result in the original
request being cancelled. We are hitting the above check and error with
v5.0.0 in OpenStack CI that made me wonder if we are hitting the
original issue in this bug where QEMU is < v5.0.0?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-10-01 Thread Lee Yarwood

> @Christian Yes we can test this in OpenStack CI with PPAs if you 
> could provide Bionic based builds.

I'm not sure what I was thinking here but we *can* use Focal based
builds of QEMU in CI now.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.config \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-30 Thread Lee Yarwood

@Christian Yes we can test this in OpenStack CI with PPAs if you could
provide Bionic based builds.

Better yet I could even try to setup a git bisect locally if you have
steps to build from a source tree somewhere?

@James Did you try using OpenStack CI sized hosts (8 vCPUs, 8GB of RAM)
and the QEMU virt_type? That was the most reliable way I found of
hitting this and running the `tox -e full` tempest target to envoke a
full tempest run.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  Incomplete

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-29 Thread Christian Ehrhardt 

Thanks James, but now I'm  unsure where to go from here as it isn't
reproducible with many tries at different scales that James and I did.

@Sean/Lee
Since you wondered if it might be due to Ubuntu Delta on top of 4.2 - there are 
two things we could compare Ubuntu's qemu to then:
1. qemu 4.2 as released by upstream
2. qemu 4.2 as build in centos (they have some delta as well)
Not sure I can provide you #2 easily and #1 will always need a bit of delta to 
build correctly. All of that would be doable still, but in general if 
I'd provide you PPA builds could you try those in your environment to 
!reliably! trigger the issue allowing us to play bisect-ping-pong? The word 
"reliable" is important here as we'd need to sort out builds/patches by a 
reliable yes/no on each step.

@Sean/Lee
- which "the centos 8 build of the same qemu" version is that exactly? I might 
take a look at comparing the patches applied. We are at 4.2.1 already, the 
latest I found there was 4.2.0-29.el8.3.x86_64.rpm which already is the 
advanced-virt version (otherwise 2.12 based).

@Sean/Lee
All of the following suggested approaches depend on the question if you can 
test this reliably with different qemu PPA builds:
- qemu 4.2 (as upstream) vs usual Ubuntu build -> find the offending patch in 
our delta
- test Ubuntu 20.10 which has qemu 5.0 and libvirt 6.6 -> if fixed there find 
by which change
- qemu 4.2 Ubuntu vs qemu 4.2 as in CentOS (but build for Ubuntu) -> if the 
latter works better then let us find by which (set of) patches.


I was checking the Delta on Centos8 advanced-virt qemu 4.2 as that was reported
to (maybe) work better. I was comparing which patches are applied, that are no
on Ubuntu and which of those might be related.
Among several individual fixes for some issues the biggest patch sets
are feature backports for Enhanced LUKS/backup/snapshot handling,
multifd migration/cancel, block-mirror, HMAT changes, virtiofs, qemu-img zero
write, arm time handling and some related build time self tests.
Due to the nature of these changes some affect the block handling by affecting 
block/job/hotplug. But they might only do so by accident, nothing is clearly 
for addressing the issue that
was reported. And even of the list below most seem unrelated - so as Sean 
assumed maybe it just isn't exercised on centos enough to be seen there?

eca0f3524a4eb57d03a56b0cbcef5527a0981ce4 backup: don't acquire aio_context in 
backup_clean
58226634c4b02af7b10862f7fbd3610a344bfb7f backup: Improve error for 
bdrv_getlength() failure
958a04bd32af18d9a207bcc78046e56a202aebc2 backup: Make sure that source and 
target size match
7b8e4857426f2e2de2441749996c6161b550bada block: Add flags to 
bdrv(_co)_truncate()
92b92799dc8662b6f71809100a4aabc1ae408ebb block: Add flags to 
BlockDriver.bdrv_co_truncate()
087ab8e775f48766068e65de1bc99d03b40d1670 block: always fill entire LUKS header 
space with zeros
8c6242b6f383e43fd11d2c50f8bcdd2bba1100fc block-backend: Add flags to 
blk_truncate()
564806c529d4e0acad209b1e5b864a8886092f1f block-backend: Reorder flush/pdiscard 
function definitions
0abf2581717a19d9749d5c2ff8acd0ac203452c2 block/backup-top: Don't acquire 
context while dropping top
1de6b45fb5c1489b450df7d1a4c692bba9678ce6 block: bdrv_reopen() with backing file 
in different AioContext
e1d7f8bb1ec0c6911dcea81641ce6139dbded02d block.c: adding bdrv_co_delete_file
69032253c33ae1774233c63cedf36d32242a85fc block/curl: HTTP header field names 
are case insensitive
7788a319399f17476ff1dd43164c869e320820a2 block/curl: HTTP header fields allow 
whitespace around values
91005a495e228ebd7e5e173cd18f952450eef82d blockdev: Acquire AioContext on dirty 
bitmap functions
471ded690e19689018535e3f48480507ed073e22 blockdev: fix coding style issues in 
drive_backup_prepare
3ea67e08832775a28d0bd2795f01bc77e7ea1512 blockdev: honor 
bdrv_try_set_aio_context() context requirements
c6996cf9a6c759c29919642be9a73ac64b38301b blockdev: Promote several bitmap 
functions to non-static
377410f6fb4f6b0d26d4a028c20766fae05de17e blockdev: Return bs to the proper 
context on snapshot abort
bb4e58c6137e80129b955789dd4b66c1504f20dc blockdev: Split off basic bitmap 
operations for qemu-img
5b7bfe515ecbd584b40ff6e41d2fd8b37c7d5139 blockdev: unify qmp_blockdev_backup 
and blockdev-backup transaction paths
2288ccfac96281c316db942d10e3f921c1373064 blockdev: unify qmp_drive_backup and 
drive-backup transaction paths
7f16476fab14fc32388e0ebae793f64673848efa block: Fix blk->in_flight during 
blk_wait_while_drained()
30dd65f307b647eef8156c4a33bd007823ef85cb block: Fix cross-AioContext 
blockdev-snapshot
eeea1faa099f82328f5831cf252f8ce0a59a9287 block: Fix leak in 
bdrv_create_file_fallback()
fd17146cd93d1704cd96d7c2757b325fc7aac6fd block: Generic file creation fallback
fbb92b6798894d3bf62fe3578d99fa62c720b242 block: Increase BB.in_flight for 
coroutine and sync interfaces
17e1e2be5f9e84e0298e28e70675655b43e225ea block: Introduce 
'bdrv_reopen_commit_post' step

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-25 Thread James Page

I'm not able to reproduce at this point in time - as the version of
libvirt/qemu is the same I deployed an Ussuri cloud on focal and ran
block device add/remove with concurrency for several hundred iterations
but I was not able to trigger a detach failure.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-86-instance-003a/.config \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-24 Thread Christian Ehrhardt 

As outlined before the time from device_del to the DEVICE_DELETED is
indeed "a bit longer" in Focal (from ~0.1s to ~6s), but other than that
I couldn't find anything else yet.

We were wondering due to [1] in a related bug if it might be dependent to 
(over)load.
I was consuming Disk and CPU in a controlled fashion (stress-ng --cpu 8 --io 4 
--hdd 4 1x in the container the load is in and 1x on the Host). On that I was 
running attach/detach loops but in all of a two hundred retries it was 
"->device_del <-DEVICE_DELETED <-DEVICE_DELETED ->blockdev-del ->blockdev-del".

James Page was doing tests (thanks) with real Openstack and Ceph but
couldn't - so far -trigger the issue either. He will likely later update
the bug as well, but is currently trying to ramp up concurrency to more
likely hit the issue.

https://bugs.launchpad.net/nova/+bug/1882521/comments/8

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-21 Thread Christian Ehrhardt 

Hi,
interesting bug report - thanks lee, Kashyap and Sean - as well as Danil for 
taking a look already. 

If this would always fail no unplugs would work ever which I knew can't
be right as I test that. So we need to find what is different...


@Openstack people - is that reliably triggering at the very first hot-detach 
for you or more like "happening in one of a million detaches triggered by the 
CI"?


First I simplified the case from "massive openstack CI" to just a few commands 
and small XMLs...
So I tried to cut ceph out of the equation here and compared hot remove between 
Bionic and Focal qemu/libvirt when removing a local image file.

$ qemu-img create -f qcow2 /var/lib/uvtool/libvirt/images/testdisk.qcow 20m
Disk:

  
  
  


The systems have just this guest each and the log debug config is like:
 /etc/libvirt/libvirtd.conf:
 > log_filters="3:qemu 3:libvirt 4:object 2:json 4:event 3:util"  
 > log_outputs="2:file:/var/log/libvirtd.log"
 $ rm /var/log/libvirtd.log
 $ systemctl restart libvirtd

Detach via:
 $ virsh detach-disk f-on-b vdc --live

I compared the monitor streams in those two cases but they both deliver 
DEVICE_DELETED for the 
/machine/peripheral/virtio-disk2/virtio-backend as well as 
/machine/peripheral/virtio-disk2.


@Jamespage could you get me a system with at least one auxiliary ceph disk that 
I can attach/detach to check if this might be in any way ceph specific?


Details in case we need them for comparison later on ...:

## Bionic ##

libvirt checks devices before removal:

2020-09-21 11:00:19.287+: 5755: info : qemuMonitorSend:1076 : 
QEMU_MONITOR_SEND_MSG: mon=0x7fdf94001430 
msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-12"}
2020-09-21 11:00:19.288+: 5686: info : qemuMonitorJSONIOProcessLine:213 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7fdf94001430 reply={"return": [{"name": 
"virtio-disk2", "type": "child"}, {"name": "virtio-disk1", 
"type": "child"}, {"name": "virtio-disk0", "type": 
"child"}, {"name": "video0", "type": "child"}, 
{"name": "serial0", "type": "child"}, {"name": "balloon0", "type": 
"child"}, {"name": "net0", "type": 
"child"}, {"name": "usb", "type": "child"}, 
{"name": "type", "type": "string"}], "id": "libvirt-12"}

Then send removal command

2020-09-21 11:00:48.672+: 5691: info : qemuMonitorSend:1076 : 
QEMU_MONITOR_SEND_MSG: mon=0x7fdf94001430 
msg={"execute":"device_del","arguments":{"id":"virtio-disk2"},"id":"libvirt-13"}
2020-09-21 11:00:48.673+: 5686: info : qemuMonitorJSONIOProcessLine:213 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7fdf94001430 reply={"return": {}, "id": 
"libvirt-13"}
020-09-21 11:00:48.721+: 5686: info : qemuMonitorJSONIOProcessLine:208 : 
QEMU_MONITOR_RECV_EVENT: mon=0x7fdf94001430 event={"timestamp": {"seconds": 
1600686048, "microseconds": 720908}, "event": "DEVICE_DELETED", "data": 
{"path": "/machine/peripheral/virtio-disk2/virtio-backend"}}
2020-09-21 11:00:48.723+: 5686: info : qemuMonitorJSONIOProcessLine:208 : 
QEMU_MONITOR_RECV_EVENT: mon=0x7fdf94001430 event={"timestamp": {"seconds": 
1600686048, "microseconds": 723091}, "event": "DEVICE_DELETED", "data": 
{"device": "virtio-disk2", "path": "/machine/peripheral/virtio-disk2"}}

^^ both DEVICE_DELETED occur in response to the "device_del".

vv Now also remove the device, but that already happened implicitly

2020-09-21 11:00:48.723+: 5691: info : qemuMonitorSend:1076 : 
QEMU_MONITOR_SEND_MSG: mon=0x7fdf94001430 
msg={"execute":"human-monitor-command","arguments":{"command-line":"drive_del 
drive-virtio-disk2"},"id":"libvirt-14"}
2020-09-21 11:00:48.724+: 5686: info : qemuMonitorJSONIOProcessLine:213 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7fdf94001430 reply={"return": "Device 
'drive-virtio-disk2' not found\r\n", "id": "libvirt-14"}

In the follow up query virtio-disk2 is removed

2020-09-21 11:00:48.875+: 5691: info : qemuMonitorSend:1076 : 
QEMU_MONITOR_SEND_MSG: mon=0x7fdf94001430 
msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-15"}
2020-09-21 11:00:48.876+: 5686: info : qemuMonitorJSONIOProcessLine:213 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7fdf94001430 reply={"return": [{"name": 
"virtio-disk1", "type": "child"}, {"name": "virtio-disk0", 
"type": "child"}, {"name": "video0", "type": 
"child"}, {"name": "serial0", "type": "child"}, 
{"name": "balloon0", "type": "child"}, {"name": "net0", 
"type": "child"}, {"name": "usb", "type": 
"child"}, {"name": "type", "type": "string"}], "id": 
"libvirt-15"}


## Focal ##

libvirt checks devices before removal:

2020-09-21 11:00:20.856+: 6395: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f463800b3e0 
msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-12"}
2020-09-21 11:00:20.857+: 6319: info : qemuMonitorJSONIOProcessLine:239 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7f463800b3e0 reply={"return": [{"name": "type", 
"type": "string"}, {"name": "pci.7", "type": "child"},

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-16 Thread sean mooney

hum i was hoping to indicate this affect focal in some what but not sure how to 
do that but this issue
happens with the ubunutu 20.04 version of qemu 4.2
it does not seam to happen with the centos 8 build of the same qemu so i dont 
know if there is a delta in packages or if its just a case that in the 
openstack ci we have more testing based on ubuntu 20.04 and its appearing there 
more as a result.


** Also affects: ubuntu
   Importance: Undecided
   Status: New

** Package changed: ubuntu => qemu (Ubuntu)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New
Status in qemu package in Ubuntu:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Daniel Berrange

Note that there are many VMs running, so it is important to isolate logs
for just one of them, by matching on  mon=0x.

For example considering this

$ grep 0x7f021c0b9c70 libvirtd.log  | grep -E
'QEMU_MONITOR_(RECV_REPLY|RECV_EVENT|SEND_MSG)'

We can see the "device_del" request:

2020-09-03 19:53:47.114+: 65331: info : qemuMonitorSend:993 : 
QEMU_MONITOR_SEND_MSG: mon=0x7f021c0b9c70 
msg={"execute":"device_del","arguments":{"id":"virtio-disk1"},"id":"libvirt-367"}
2020-09-03 19:53:47.116+: 65328: info : qemuMonitorJSONIOProcessLine:239 : 
QEMU_MONITOR_RECV_REPLY: mon=0x7f021c0b9c70 reply={"return": {}, "id": 
"libvirt-367"}


and then the first DEVICE_DELETED event:

2020-09-03 19:53:52.539+: 65328: info :
qemuMonitorJSONIOProcessLine:234 : QEMU_MONITOR_RECV_EVENT:
mon=0x7f021c0b9c70 event={"timestamp": {"seconds": 1599162832,
"microseconds": 539721}, "event": "DEVICE_DELETED", "data": {"path":
"/machine/peripheral/virtio-disk1/virtio-backend"}}


The second, important, DEVICE_DELETED event never arrives.  OpenStack 
hopelessly just retries  device_del forever more.

Note in normal case the first DEVICE_DELETED event (for
"/machine/peripheral/virtio-disk1/virtio-backend") gets emitted from
VCPU thread context. The second DEVICE_DELETED event (for
"/machine/peripheral/virtio-disk1/")  is emitted from the RCU thread,
and that one is missing here. THe second one is the event that libvirt
needs in order to determine that unplug is complete.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Kashyap Chamarthy

Attaching the QMP requests and responses filtered out from the giant
libvirtd.log.

This file was generated using:

$> grep -Ei '(QEMU_MONITOR_SEND_MSG|QEMU_MONITOR_RECV_)' libvirtd.log |& 
   tee QMP_requests_and_replies_from_libvirtd.log

(And then I `gzip`ed it; compressed - 3.1MB; uncompressed -39MB)

** Attachment added: "QMP_requests_and_replies_from_libvirtd.log.gz"
   
https://bugs.launchpad.net/qemu/+bug/1894804/+attachment/5408622/+files/QMP_requests_and_replies_from_libvirtd.log.gz

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

2020-09-08 Thread Kashyap Chamarthy

Just to expand a little more on the "second DEVICE_DELETED" event:

Apparently this is an "oddity" of VirtIO devices where we see _two_
DEVICE_DELETED events upon hot-unplugging a device.  So, ideally, we
should see the following two events, but we're only seeing the first
one:

(1) "event": "DEVICE_DELETED", "data": {"path": 
"/machine/peripheral/virtio-disk1/virtio-backend"}}
 
(2) "event": "DEVICE_DELETED", "data": {"device": "virtio-disk1", "path": 
"/machine/peripheral/virtio-disk1"}}

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1894804

Title:
  Second DEVICE_DELETED event missing during virtio-blk disk device
  detach

Status in QEMU:
  New

Bug description:
  We are in the process of moving OpenStack CI across to use 20.04 Focal
  as the underlying OS and encountering the following issue in any test
  attempting to detach disk devices from running QEMU instances.

  We can see a single DEVICE_DELETED event raised to libvirtd for the
  /machine/peripheral/virtio-disk1/virtio-backend device but we do not
  see a second event for the actual disk. As a result the device is
  still marked as present in libvirt but QEMU reports it as missing in
  subsequent attempts to remove the device.

  The following log snippets can also be found in the following pastebin
  that's slightly easier to gork:

  http://paste.openstack.org/show/797564/

  https://review.opendev.org/#/c/746981/ libvirt: Bump
  MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION

  https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260
  nova-ceph-multistore

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/dpkg-l.txt

  ii  libvirt-daemon   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon
  ii  libvirt-daemon-driver-qemu   6.0.0-0ubuntu8.3 
 amd64Virtualization daemon QEMU connection driver
  ii  libvirt-daemon-system6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files
  ii  libvirt-daemon-system-systemd6.0.0-0ubuntu8.3 
 amd64Libvirt daemon configuration files (systemd)
  ii  libvirt-dev:amd646.0.0-0ubuntu8.3 
 amd64development files for the libvirt library
  ii  libvirt0:amd64   6.0.0-0ubuntu8.3 
 amd64library for interfacing with different virtualization systems
  [..]
  ii  qemu-block-extra:amd64   1:4.2-3ubuntu6.4 
 amd64extra block backend modules for qemu-system and qemu-utils
  ii  qemu-slof20191209+dfsg-1  
 all  Slimline Open Firmware -- QEMU PowerPC version
  ii  qemu-system  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries
  ii  qemu-system-arm  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (arm)
  ii  qemu-system-common   1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (common files)
  ii  qemu-system-data 1:4.2-3ubuntu6.4 
 all  QEMU full system emulation (data files)
  ii  qemu-system-mips 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (mips)
  ii  qemu-system-misc 1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (miscellaneous)
  ii  qemu-system-ppc  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (ppc)
  ii  qemu-system-s390x1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (s390x)
  ii  qemu-system-sparc1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (sparc)
  ii  qemu-system-x86  1:4.2-3ubuntu6.4 
 amd64QEMU full system emulation binaries (x86)
  ii  qemu-utils   1:4.2-3ubuntu6.4 
 amd64QEMU utilities

  
https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/controller/logs/libvirt/qemu
  /instance-003a_log.txt

  2020-09-07 19:29:55.021+: starting up libvirt version: 6.0.0, package: 
0ubuntu8.3 (Marc Deslauriers  Thu, 30 Jul 2020 
06:40:28 -0400), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.4, kernel: 
5.4.0-45-generic, hostname: ubuntu-focal-ovh-bhs1-0019682147
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/var/lib/libvirt/qemu/domain-86-instance-003a \

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

[Bug 1894804] Re: Second DEVICE_DELETED event missing during virtio-blk disk device detach

18 matches

Site Navigation

Mail list logo

Footer information