[Yahoo-eng-team] [Bug 1921306] Re: When Instance is paused state, live migration failure.

[Lee Yarwood]

This smells like a known QEMU issue that we encountered in CI a while
ago causing the underlying guest to crash:

[QEMU] Back-n-forth live migration of a paused VM results in QEMU crash
with: "bdrv_inactivate_recurse: Assertion `!(bs->open_flags &
BDRV_O_INACTIVE)' failed."

https://bugzilla.redhat.com/show_bug.cgi?id=1713009

You should see the above error captured in the guest log under
/var/log/libvirt/qemu/$domain_name.log

Marking this bug as invalid for Nova but feel free to add the QEMU
project in assuming you're using the Ubuntu provided package.

** Bug watch added: Red Hat Bugzilla #1713009
   https://bugzilla.redhat.com/show_bug.cgi?id=1713009

** Changed in: nova
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921306

Title:
  When Instance is paused state,live migration failure.

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  When the virtual machine is in the paused state and continuously live
  migrated twice, the Instance state changes to ERROR. The version of
  nova is Rocky。

  2021-03-22 19:34:38.388 6 DEBUG nova.virt.libvirt.guest [- req-None - - - - 
-] Failed to get job stats: Unable to read from monitor: Connection reset by 
peer get_job_info 
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:767
  2021-03-22 19:34:38.389 6 WARNING nova.virt.libvirt.driver [- req-None - - - 
- -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Error monitoring 
migration: Unable to read from monitor: Connection reset by peer: libvirtError: 
Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7793, in _live_migration
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] finish_event, disk_paths)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7593, in _live_migration_monitor
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] info = guest.get_job_info()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", 
line 751, in get_job_info
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] stats = self._domain.jobStats()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, 
in doit
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] result = proxy_call(self._autowrap, 
f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, 
in proxy_call
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = execute(f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, 
in execute
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] six.reraise(c, e, tb)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, 
in tworker
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = meth(*args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] if ret is None: raise libvirtError 
('virDomainGetJobStats() failed', dom=self)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from 
monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR 

[Yahoo-eng-team] [Bug 1921306] Re: When Instance is paused state, live migration failure.

[Pierre Thévenet]

** Changed in: nova
   Status: Expired => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921306

Title:
  When Instance is paused state,live migration failure.

Status in OpenStack Compute (nova):
  New

Bug description:
  When the virtual machine is in the paused state and continuously live
  migrated twice, the Instance state changes to ERROR. The version of
  nova is Rocky。

  2021-03-22 19:34:38.388 6 DEBUG nova.virt.libvirt.guest [- req-None - - - - 
-] Failed to get job stats: Unable to read from monitor: Connection reset by 
peer get_job_info 
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:767
  2021-03-22 19:34:38.389 6 WARNING nova.virt.libvirt.driver [- req-None - - - 
- -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Error monitoring 
migration: Unable to read from monitor: Connection reset by peer: libvirtError: 
Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7793, in _live_migration
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] finish_event, disk_paths)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7593, in _live_migration_monitor
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] info = guest.get_job_info()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", 
line 751, in get_job_info
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] stats = self._domain.jobStats()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, 
in doit
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] result = proxy_call(self._autowrap, 
f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, 
in proxy_call
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = execute(f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, 
in execute
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] six.reraise(c, e, tb)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, 
in tworker
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = meth(*args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] if ret is None: raise libvirtError 
('virDomainGetJobStats() failed', dom=self)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from 
monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]
  2021-03-22 19:34:38.394 6 DEBUG nova.virt.libvirt.driver [- req-None - - - - 
-] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration monitoring 
is all done _live_migration 
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7800
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [- req-None - - - - -] 
[instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration failed.: 
libvirtError: Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] 

[Yahoo-eng-team] [Bug 1921306] Re: When Instance is paused state, live migration failure.

[Launchpad Bug Tracker]

[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921306

Title:
  When Instance is paused state,live migration failure.

Status in OpenStack Compute (nova):
  Expired

Bug description:
  When the virtual machine is in the paused state and continuously live
  migrated twice, the Instance state changes to ERROR. The version of
  nova is Rocky。

  2021-03-22 19:34:38.388 6 DEBUG nova.virt.libvirt.guest [- req-None - - - - 
-] Failed to get job stats: Unable to read from monitor: Connection reset by 
peer get_job_info 
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:767
  2021-03-22 19:34:38.389 6 WARNING nova.virt.libvirt.driver [- req-None - - - 
- -] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Error monitoring 
migration: Unable to read from monitor: Connection reset by peer: libvirtError: 
Unable to read from monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] Traceback (most recent call last):
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7793, in _live_migration
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] finish_event, disk_paths)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", 
line 7593, in _live_migration_monitor
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] info = guest.get_job_info()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", 
line 751, in get_job_info
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] stats = self._domain.jobStats()
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, 
in doit
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] result = proxy_call(self._autowrap, 
f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, 
in proxy_call
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = execute(f, *args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, 
in execute
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] six.reraise(c, e, tb)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, 
in tworker
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] rv = meth(*args, **kwargs)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]   File 
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1433, in jobStats
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] if ret is None: raise libvirtError 
('virDomainGetJobStats() failed', dom=self)
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596] libvirtError: Unable to read from 
monitor: Connection reset by peer
  2021-03-22 19:34:38.389 6 ERROR nova.virt.libvirt.driver [instance: 
d1d2af1f-e973-438c-a6b6-b628091d3596]
  2021-03-22 19:34:38.394 6 DEBUG nova.virt.libvirt.driver [- req-None - - - - 
-] [instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration monitoring 
is all done _live_migration 
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:7800
  2021-03-22 19:34:38.394 6 ERROR nova.compute.manager [- req-None - - - - -] 
[instance: d1d2af1f-e973-438c-a6b6-b628091d3596] Live migration failed.: 
libvirtError: Unable to read from monitor: Connection reset by peer