Public bug reported:

Seeing this in some CI runs:

http://logs.openstack.org/67/560467/1/check/nova-
next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152

Apr 11 16:19:23.843152 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver [None req-4ca5f968-bbce-4826-b16c-cc02d63c07c2 
service nova] Failure rebasing volume /dev/sda on vdb.: libvirtError: internal 
error: unable to execute QEMU command 'block-job-complete': The active block 
job 'drive-virtio-disk1' cannot be completed
Apr 11 16:19:23.843399 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Apr 11 16:19:23.843646 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1529, in _swap_volume
Apr 11 16:19:23.843882 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
Apr 11 16:19:23.844165 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 757, in abort_job
Apr 11 16:19:23.844416 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     
self._guest._domain.blockJobAbort(self._disk, flags=flags)
Apr 11 16:19:23.844666 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
Apr 11 16:19:23.844905 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, 
*args, **kwargs)
Apr 11 16:19:23.845168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in 
proxy_call
Apr 11 16:19:23.845412 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
Apr 11 16:19:23.845659 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 11 16:19:23.845909 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
Apr 11 16:19:23.846168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 11 16:19:23.846445 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
Apr 11 16:19:23.846707 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
Apr 11 16:19:23.846960 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError 
('virDomainBlockJobAbort() failed', dom=self)
Apr 11 16:19:23.878201 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver libvirtError: internal error: unable to execute 
QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' 
cannot be completed
Apr 11 16:19:23.878506 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: 
ERROR nova.virt.libvirt.driver 

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20swap%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20block%20copy%20still%20active%3A%20domain%20has%20active%20block%20job%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

7 hits in 7 days, check and gate, all failures.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt swap-volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1763410

Title:
  swap volume intermittently fails with "libvirtError: block copy still
  active: domain has active block job"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seeing this in some CI runs:

  http://logs.openstack.org/67/560467/1/check/nova-
  next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152

  Apr 11 16:19:23.843152 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver [None 
req-4ca5f968-bbce-4826-b16c-cc02d63c07c2 service nova] Failure rebasing volume 
/dev/sda on vdb.: libvirtError: internal error: unable to execute QEMU command 
'block-job-complete': The active block job 'drive-virtio-disk1' cannot be 
completed
  Apr 11 16:19:23.843399 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver Traceback (most recent call 
last):
  Apr 11 16:19:23.843646 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1529, in _swap_volume
  Apr 11 16:19:23.843882 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     
dev.abort_job(pivot=True)
  Apr 11 16:19:23.844165 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 757, in abort_job
  Apr 11 16:19:23.844416 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     
self._guest._domain.blockJobAbort(self._disk, flags=flags)
  Apr 11 16:19:23.844666 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  Apr 11 16:19:23.844905 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     result = 
proxy_call(self._autowrap, f, *args, **kwargs)
  Apr 11 16:19:23.845168 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in 
proxy_call
  Apr 11 16:19:23.845412 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, 
**kwargs)
  Apr 11 16:19:23.845659 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  Apr 11 16:19:23.845909 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
  Apr 11 16:19:23.846168 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  Apr 11 16:19:23.846445 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     rv = meth(*args, 
**kwargs)
  Apr 11 16:19:23.846707 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver   File 
"/usr/local/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
  Apr 11 16:19:23.846960 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver     if ret == -1: raise 
libvirtError ('virDomainBlockJobAbort() failed', dom=self)
  Apr 11 16:19:23.878201 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver libvirtError: internal 
error: unable to execute QEMU command 'block-job-complete': The active block 
job 'drive-virtio-disk1' cannot be completed
  Apr 11 16:19:23.878506 ubuntu-xenial-inap-mtl01-0003447058 
nova-compute[31776]: ERROR nova.virt.libvirt.driver 

  
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20swap%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20block%20copy%20still%20active%3A%20domain%20has%20active%20block%20job%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

  7 hits in 7 days, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1763410/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to