Public bug reported: Description ===========
When creating a server, if a compute node doesn't belong any cell and nova-scheduler selects the host, creating a server fails (HostMappingNotFound) and a server status become stuck in 'BUILD'. The server cannot be deleted by "Delete server" API. This issue can be avoided by executing the following command before creating a server. nova-manage cell_v2 map_cell_and_hosts But an error handling(*1) for HostMappingNotFound exists. So it should be executed successfully. *1: https://github.com/openstack/nova/blob/8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac/nova/conductor/manager.py#L943-L945 Steps to reproduce ================== 1. start nova processes (except nova-compute) and other components processes (galnce, neutorn, keystone, cinder, etc) in devstack 2. start nova-compute process 3. create a server (POST /servers) Expected result =============== The status of the server becomes ERROR. The server can be deleted. Actual result ============= The server becomes stuck in 'BUILD' status. The server cannot be deleted. Environment =========== nova: master (commit 8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac) devstack: master (commit 9b2a2fa55dbba724a781d2720546611a8add893) OS: Ubuntu 16.04.2 LTS Logs & Configs ============== stack@devstack-master:/opt/devstack$ nova boot --flavor 1 --image 91bd1f63-77ff-41d9-abe3-367163ef9673 server1 +--------------------------------------+-----------------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | server1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-8lxtcp9q | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | V3FLw3BDJUQu | | config_drive | | | created | 2017-05-26T07:35:12Z | | description | - | | flavor | m1.tiny (1) | | hostId | | | host_status | | | id | 421bb801-005a-4f73-9fe1-0916fd691b2b | | image | cirros-0.3.5-x86_64-disk (91bd1f63-77ff-41d9-abe3-367163ef9673) | | key_name | - | | locked | False | | metadata | {} | | name | server1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tags | [] | | tenant_id | 56eb461553af4c28a09934bd3f1d6769 | | updated | 2017-05-26T07:35:12Z | | user_id | c001d3de4c1d49e18d3b23faf1bcebc2 | +--------------------------------------+-----------------------------------------------------------------+ stack@devstack-master:/opt/devstack$ nova list +--------------------------------------+---------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+----------+ | 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | | +--------------------------------------+---------+--------+------------+-------------+----------+ stack@devstack-master:/opt/devstack$ nova delete server1 Request to delete server server1 has been accepted. stack@devstack-master:/opt/devstack$ nova list +--------------------------------------+---------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+----------+ | 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | | +--------------------------------------+---------+--------+------------+-------------+----------+ nova-conductor log: ****************************************************** {{(pid=9578) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2797}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Creating RPC server for service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:166}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Creating RPC server for service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:166}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Join ServiceGroup membership for this service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:184}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9955) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Join ServiceGroup membership for this service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:184}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9954) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING oslo_config.cfg [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Option "scheduler_default_filters" from group "DEFAULT" is deprecated. Use option "enabled_filters" from group "filter_scheduler". 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to schedule instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 916, in schedule_and_build_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict()) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 637, in _schedule_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/utils.py", line 371, in wrapped 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 51, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 37, in __run_method 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/query.py", line 32, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 130, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=self.retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager timeout=timeout, retry=retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise result 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound_Remote: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager selected_hosts = self._schedule(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self._get_all_host_states(elevated, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._get_host_states(context, compute_nodes, services) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager self._get_instance_info(context, compute)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager inst_dict = self._get_instances_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hm = objects.HostMapping.get_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = fn(cls, context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager db_mapping = cls._get_by_host_from_db(context, host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return fn(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise exception.HostMappingNotFound(name=host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to compute_task_build_instances: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner 5月 26 16:35:12 devstack-master nova-conductor[9578]: return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: dests = self.driver.select_destinations(ctxt, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: selected_hosts = self._schedule(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule 5月 26 16:35:12 devstack-master nova-conductor[9578]: hosts = self._get_all_host_states(elevated, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids 5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._get_host_states(context, compute_nodes, services) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: self._get_instance_info(context, compute)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info 5月 26 16:35:12 devstack-master nova-conductor[9578]: inst_dict = self._get_instances_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: hm = objects.HostMapping.get_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: result = fn(cls, context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: db_mapping = cls._get_by_host_from_db(context, host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: return fn(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 5月 26 16:35:12 devstack-master nova-conductor[9578]: raise exception.HostMappingNotFound(name=host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: HostMappingNotFound: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] [instance: 421bb801-005a-4f73-9fe1-0916fd691b2b] Setting instance to ERROR state. 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Exception during message handling 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server build_requests=build_requests) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 890, in _bury_in_cell0 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server exc, legacy_spec) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 355, in _set_vm_state_and_notify 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ex, request_spec) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/utils.py", line 103, in set_vm_state_and_notify 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server instance.save() 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/objects/instance.py", line 781, in save 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=_expected_cols(expected_attrs)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/api.py", line 860, in instance_update_and_get_original 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server expected=expected) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server self.force_reraise() 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2673, in instance_update_and_get_original 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=columns_to_join) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1929, in _instance_get_by_uuid 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server raise exception.InstanceNotFound(instance_id=uuid) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 421bb801-005a-4f73-9fe1-0916fd691b2b could not be found. 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ** Affects: nova Importance: Undecided Assignee: Takashi NATSUME (natsume-takashi) Status: New ** Tags: cells conductor -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1693721 Title: Error handling for HostMappingNotFound in conductor manager does not work properly Status in OpenStack Compute (nova): New Bug description: Description =========== When creating a server, if a compute node doesn't belong any cell and nova-scheduler selects the host, creating a server fails (HostMappingNotFound) and a server status become stuck in 'BUILD'. The server cannot be deleted by "Delete server" API. This issue can be avoided by executing the following command before creating a server. nova-manage cell_v2 map_cell_and_hosts But an error handling(*1) for HostMappingNotFound exists. So it should be executed successfully. *1: https://github.com/openstack/nova/blob/8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac/nova/conductor/manager.py#L943-L945 Steps to reproduce ================== 1. start nova processes (except nova-compute) and other components processes (galnce, neutorn, keystone, cinder, etc) in devstack 2. start nova-compute process 3. create a server (POST /servers) Expected result =============== The status of the server becomes ERROR. The server can be deleted. Actual result ============= The server becomes stuck in 'BUILD' status. The server cannot be deleted. Environment =========== nova: master (commit 8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac) devstack: master (commit 9b2a2fa55dbba724a781d2720546611a8add893) OS: Ubuntu 16.04.2 LTS Logs & Configs ============== stack@devstack-master:/opt/devstack$ nova boot --flavor 1 --image 91bd1f63-77ff-41d9-abe3-367163ef9673 server1 +--------------------------------------+-----------------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | server1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-8lxtcp9q | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | V3FLw3BDJUQu | | config_drive | | | created | 2017-05-26T07:35:12Z | | description | - | | flavor | m1.tiny (1) | | hostId | | | host_status | | | id | 421bb801-005a-4f73-9fe1-0916fd691b2b | | image | cirros-0.3.5-x86_64-disk (91bd1f63-77ff-41d9-abe3-367163ef9673) | | key_name | - | | locked | False | | metadata | {} | | name | server1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tags | [] | | tenant_id | 56eb461553af4c28a09934bd3f1d6769 | | updated | 2017-05-26T07:35:12Z | | user_id | c001d3de4c1d49e18d3b23faf1bcebc2 | +--------------------------------------+-----------------------------------------------------------------+ stack@devstack-master:/opt/devstack$ nova list +--------------------------------------+---------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+----------+ | 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | | +--------------------------------------+---------+--------+------------+-------------+----------+ stack@devstack-master:/opt/devstack$ nova delete server1 Request to delete server server1 has been accepted. stack@devstack-master:/opt/devstack$ nova list +--------------------------------------+---------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+----------+ | 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | | +--------------------------------------+---------+--------+------------+-------------+----------+ nova-conductor log: ****************************************************** {{(pid=9578) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2797}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Creating RPC server for service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:166}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Creating RPC server for service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:166}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Join ServiceGroup membership for this service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:184}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9955) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Join ServiceGroup membership for this service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:184}} 5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9954) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING oslo_config.cfg [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Option "scheduler_default_filters" from group "DEFAULT" is deprecated. Use option "enabled_filters" from group "filter_scheduler". 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to schedule instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 916, in schedule_and_build_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict()) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 637, in _schedule_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/utils.py", line 371, in wrapped 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 51, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 37, in __run_method 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/query.py", line 32, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 130, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=self.retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager timeout=timeout, retry=retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=retry) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise result 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound_Remote: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager selected_hosts = self._schedule(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self._get_all_host_states(elevated, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._get_host_states(context, compute_nodes, services) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager self._get_instance_info(context, compute)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager inst_dict = self._get_instances_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hm = objects.HostMapping.get_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = fn(cls, context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager db_mapping = cls._get_by_host_from_db(context, host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return fn(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise exception.HostMappingNotFound(name=host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}} 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to compute_task_build_instances: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner 5月 26 16:35:12 devstack-master nova-conductor[9578]: return func(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: dests = self.driver.select_destinations(ctxt, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations 5月 26 16:35:12 devstack-master nova-conductor[9578]: selected_hosts = self._schedule(context, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule 5月 26 16:35:12 devstack-master nova-conductor[9578]: hosts = self._get_all_host_states(elevated, spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: spec_obj) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids 5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._get_host_states(context, compute_nodes, services) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states 5月 26 16:35:12 devstack-master nova-conductor[9578]: self._get_instance_info(context, compute)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info 5月 26 16:35:12 devstack-master nova-conductor[9578]: inst_dict = self._get_instances_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: hm = objects.HostMapping.get_by_host(context, host_name) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: result = fn(cls, context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host 5月 26 16:35:12 devstack-master nova-conductor[9578]: db_mapping = cls._get_by_host_from_db(context, host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: return fn(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 5月 26 16:35:12 devstack-master nova-conductor[9578]: raise exception.HostMappingNotFound(name=host) 5月 26 16:35:12 devstack-master nova-conductor[9578]: HostMappingNotFound: Host 'devstack-master' is not mapped to any cell 5月 26 16:35:12 devstack-master nova-conductor[9578]: 5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] [instance: 421bb801-005a-4f73-9fe1-0916fd691b2b] Setting instance to ERROR state. 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Exception during message handling 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server build_requests=build_requests) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 890, in _bury_in_cell0 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server exc, legacy_spec) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 355, in _set_vm_state_and_notify 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ex, request_spec) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/utils.py", line 103, in set_vm_state_and_notify 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server instance.save() 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/objects/instance.py", line 781, in save 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=_expected_cols(expected_attrs)) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/api.py", line 860, in instance_update_and_get_original 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server expected=expected) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server self.force_reraise() 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2673, in instance_update_and_get_original 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=columns_to_join) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1929, in _instance_get_by_uuid 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server raise exception.InstanceNotFound(instance_id=uuid) 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 421bb801-005a-4f73-9fe1-0916fd691b2b could not be found. 5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1693721/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
