Public bug reported: Description =========== Live-migration operation failed when _post_live_migration function executed on source compute node. The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting.
Steps to reproduce ================== * I did live-migrate an instance. * then I did "systemctl restart openstack-nova-scheduler". * Instance's vm_state is error. Expected result =============== Instance's vm_state is active, this migration is successful. Actual result ============= Instance's vm_state is error Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Pike version Logs & Configs ============== 2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/' 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last): 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] finish_event, disk_paths) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] migrate_data) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] function_name, call_dict, binary) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return f(self, context, *args, **kw) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] kwargs['instance'], e, sys.exc_info()) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return function(self, context, *args, **kwargs) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self._update_scheduler_instance_info(ctxt, instance) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] instance) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] 233 instance-00000173 running ** Affects: nova Importance: Undecided Status: New ** Description changed: Description =========== - Live-migration operation failed when _post_live_migration function executed on source compute node. + Live-migration operation failed when _post_live_migration function executed on source compute node. The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting. - + Steps to reproduce ================== - * I did live-migration an instance. - * then I did "systemctl restart openstack-nova-scheduler" - * Instance's vm_state is error + * I did live-migrate an instance. + * then I did "systemctl restart openstack-nova-scheduler". + * Instance's vm_state is error. Expected result =============== Instance's vm_state is active, this migration is successful. Actual result ============= Instance's vm_state is error Environment =========== 1. Exact version of OpenStack you are running. See the following - list for all releases: http://docs.openstack.org/releases/ + list for all releases: http://docs.openstack.org/releases/ - Pike version + Pike version Logs & Configs ============== 2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/' 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last): 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] finish_event, disk_paths) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] migrate_data) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] function_name, call_dict, binary) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return f(self, context, *args, **kw) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] kwargs['instance'], e, sys.exc_info()) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return function(self, context, *args, **kwargs) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self._update_scheduler_instance_info(ctxt, instance) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] instance) - 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] + 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] 233 instance-00000173 running -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1737700 Title: live-migration error when no exchange 'scheduler_fanout' in vhost Status in OpenStack Compute (nova): New Bug description: Description =========== Live-migration operation failed when _post_live_migration function executed on source compute node. The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting. Steps to reproduce ================== * I did live-migrate an instance. * then I did "systemctl restart openstack-nova-scheduler". * Instance's vm_state is error. Expected result =============== Instance's vm_state is active, this migration is successful. Actual result ============= Instance's vm_state is error Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Pike version Logs & Configs ============== 2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/' 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last): 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] finish_event, disk_paths) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] migrate_data) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] function_name, call_dict, binary) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return f(self, context, *args, **kw) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] kwargs['instance'], e, sys.exc_info()) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self.force_reraise() 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] six.reraise(self.type_, self.value, self.tb) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] return function(self, context, *args, **kwargs) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] self._update_scheduler_instance_info(ctxt, instance) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] instance) 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] 233 instance-00000173 running To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1737700/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
