On Fri, Nov 2, 2018 at 9:12 PM Bernd Bausch <berndbau...@gmail.com> wrote:
> This is the part that determines the hypervisor: > >> # /etc/nova/nova-compute.conf >> [DEFAULT] >> compute_driver=libvirt.LibvirtDriver >> [libvirt] >> virt_type=xen >> > > Nova uses libvirt to manage VMs, and libvirt uses Xen. According to the > Nova/Xen > configuration guide > <https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-xen-libvirt.html>, > it is enough to set these variables. However, if you have not read that > config page, do it now. Are you using correct image formats? Do you find > anything in the other logfiles mentioned on that page? > > The references to *qemu-img *in the nova-compute log don't mean that qemu > is used. Nova uses *qemu-img* as a general tool for converting VM images, > no matter which hypervisor is configured. > > However, something prevents Nova to access one of the disk images, namely > /home/caslab/xenguest1/domains/guest1/disk.img. Is *guest1* the VM you > started manually? > > I see: > > Unexpected error while running command. > Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 > --cpu=30 -- env LC_ALL=C LANG=C * qemu-img info > /home/caslab/xenguest1/domains/guest1/disk.img* > > and > > Could not open '/home/caslab/xenguest1/domains/guest1/disk.img': > Permission denied > > I don't know, though, whether these errors are related to your problem. > Why is Libvirt or Xen unable to start your instances? I suggest you look > for other errors in the nova-compute.log files. > > Bernd > > On 11/2/2018 6:30 PM, Minjun Hong wrote: > > On Fri, Nov 2, 2018 at 9:05 AM Bernd Bausch <berndbau...@gmail.com> wrote: > >> Welcome to OpenStack! I think the joy of troubleshooting it is one of its >> main selling points. >> >> The conductor log says "*libxenlight failed to create new domain*" on >> various nodes. You should check the nova-compute logs on those nodes for >> relevant messages. I suspect an ill-configured interface between Nova >> Compute and libvirt/Xen. In other words, double-check the libvirt and xen >> sections of nova.conf on those nodes. >> >> Then there is this message "*Invalid **input for field >> 'identity/password/user/password': None is not of type 'string' ....*". >> This is often caused by incompatible software. I.e. you installed Openstack >> components that don't fit together. Might also be caused by a configuration >> error. >> Or it's an ordinary bug, but it seems to occur on node 2 only. >> On 11/1/2018 11:01 PM, Minjun Hong wrote: >> >> Hi everyone. >> I'm really new to OpenStack. >> After I install essential components of OpenStack, such as Nova, >> Keystone, etc, I attempted creating an instance through Openstack command >> in the terminal. >> But a trouble has occurred. It was logged in "nova-conductor.log": >> >> 2018-11-01 22:18:59.831 2570 ERROR nova.scheduler.utils >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance: >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node2 (node >>> node2): [u'Traceback (most recent call last):\n', u' File >>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in >>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u' >>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120, >>> in _build_and_run_instance\n instance_uuid=instance.uuid, >>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: Invalid input for >>> field 'identity/password/user/password': None is not of type >>> 'string'\n\nFailed validating 'type' in >>> schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n >>> {'type': 'string'}\n\nOn >>> instance['identity']['password']['user']['password']:\n None (HTTP 400) >>> (Request-ID: req-350beee4-2fed-4645-9e21-79806d7ebfe7)\n"] >>> 2018-11-01 22:18:59.833 2570 WARNING oslo_config.cfg >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Option "os_region_name" >>> from group "placement" is deprecated. Use option "region-name" from group >>> "placement". >>> 2018-11-01 22:19:10.936 2571 ERROR nova.scheduler.utils >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance: >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node4 (node >>> node4): [u'Traceback (most recent call last):\n', u' File >>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in >>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u' >>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120, >>> in _build_and_run_instance\n instance_uuid=instance.uuid, >>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error: >>> libxenlight failed to create new domain 'instance-00000005'\n"] >>> 2018-11-01 22:19:21.783 2567 ERROR nova.scheduler.utils >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance: >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node5 (node >>> node5): [u'Traceback (most recent call last):\n', u' File >>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in >>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u' >>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120, >>> in _build_and_run_instance\n instance_uuid=instance.uuid, >>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error: >>> libxenlight failed to create new domain 'instance-00000005'\n"] >>> 2018-11-01 22:19:21.783 2567 WARNING nova.scheduler.utils >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Failed to >>> compute_task_build_instances: Exceeded maximum number of retries. Exceeded >>> max scheduling attempts 3 for instance >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception: internal error: >>> libxenlight failed to create new domain 'instance-00000005': >>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max >>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. >>> Last exception: internal error: libxenlight failed to create new domain >>> 'instance-00000005' >>> 2018-11-01 22:19:21.784 2567 WARNING nova.scheduler.utils >>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7 >>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance: >>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Setting instance to ERROR state.: >>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max >>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. >>> Last exception: internal error: libxenlight failed to create new domain >>> 'instance-00000005' >> >> >> I'm not sure which component is involved in this trouble. >> And libvirt and Xen have been successfully installed on all compute node >> I have without any problem. >> >> nickeys@node2:~$ virsh create ./test.xml >>> Domain guest1 created from ./test.xml >>> nickeys@node2:~$ virsh list >>> Id Name State >>> -------------------------- >>> 0 Domain-0 running >>> 1 guest1 running >>> nickeys@node2:~$ >> >> >> What should I check first for that issue ? >> Your hint would be big help for me. >> >> Thanks! >> >> >> _______________________________________________ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> >> > Thanks for your kind answer, Bernd. > I checked the configure files which you mentioned like it can cause the > trouble. following files are my nova.conf and nova-compute.conf of the > compute node ('node2'). > > # /etc/nova/nova.conf > > [DEFAULT] >> dhcpbridge_flagfile=/etc/nova/nova.conf >> dhcpbridge=/usr/bin/nova-dhcpbridge >> state_path=/var/lib/nova >> lock_path=/var/lock/nova >> force_dhcp_release=True >> libvirt_use_virtio_for_bridges=True >> verbose=True >> ec2_private_dns_show_ip=True >> api_paste_config=/etc/nova/api-paste.ini >> enabled_apis=osapi_compute,metadata >> transport_url = rabbit://admin:1234@10.150.3.88 >> my_ip = 10.150.21.182 >> use_neutron = True >> firewall_driver = nova.virt.firewall.NoopFirewallDriver >> [api] >> auth_strategy = keystone >> [vnc] >> enabled = True >> server_listen = 0.0.0.0 >> vncserver_proxyclient_address = 10.150.21.182 >> novncproxy_base_url = http://10.150.3.88:6080/vnc_auth.html >> [glance] >> api_servers = http://10.150.3.88:9292 >> [oslo_concurrency] >> lock_path = /var/lib/nova/tmp >> [keystone_authtoken] >> auth_url = http://10.150.3.88:5000/v3 >> memcached_servers = 10.150.3.88:11211 >> auth_type = password >> project_domain_name = default >> user_domain_name = default >> project_name = service >> username = nova >> password = <KEYSTONE_PASS> >> [placement] >> os_region_name = RegionOne >> project_domain_name = Default >> project_name = service >> auth_type = password >> user_domain_name = Default >> auth_url = http://10.150.3.88:5000/v3 >> username = placement >> password = <PLACEMENT_PASS> >> [database] >> connection=sqlite:////var/lib/nova/nova.sqlite >> [api_database] >> connection=sqlite:////var/lib/nova/nova.sqlite >> [neutron] >> url = http://10.150.3.88:9696 >> auth_url = http://10.150.3.88:5000 >> auth_type = password >> project_domain_name = default >> user_domain_name = default >> region_name = RegionOne >> project_name = service >> username = neutron >> passsword = <NEUTRON_PASS> > > > # /etc/nova/nova-compute.conf >> [DEFAULT] >> compute_driver=libvirt.LibvirtDriver >> [libvirt] >> virt_type=xen > > > And I set all the compute nodes I'm using with the same configure file > above. > Also, I installed the OpenStack components, which belong to Queens > version, following the official installation guide ( > https://docs.openstack.org/install-guide/). > > By the way, I found something weird in '/var/log/nova/nova-compute.log' of > the 'node2'. > After I manually created a VM by 'virsh' command (i.g. $ virsh create > test.xml), nova-compute of 'node2' started to complain. > And on an other node which have not created a virtual machine by the > 'virsh' command, there is not below error. > I guess that the nova-compute component was configured to qemu as > virtualization type, but I have no idea where the setting value is. > > 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager >> [req-ec324126-6dfc-4011-9f45-41d014e6f900 - - - - -] Error updating >> resources for node node2.: InvalidDiskInfo: Disk info file is invalid: >> qemu-img failed to execute on >> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while >> running command. >> Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 >> --cpu=30 -- env LC_ALL=C LANG=C qemu-img info >> /home/caslab/xenguest1/domains/guest1/disk.img >> Exit code: 1 >> Stdout: u'' >> Stderr: u"qemu-img: Could not open >> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open >> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n" >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Traceback (most >> recent call last): >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7284, in >> update_available_resource_for_node >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager >> rt.update_available_resource(context, nodename) >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line >> 664, in update_available_resource >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager resources = >> self.driver.get_available_resource(nodename) >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6440, >> in get_available_resource >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager >> disk_over_committed = self._get_disk_over_committed_size_total() >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8019, >> in _get_disk_over_committed_size_total >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager config, >> block_device_info) >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7918, >> in _get_instance_disk_info_from_config >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager dk_size = >> disk_api.get_allocated_disk_size(path) >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in >> get_allocated_disk_size >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager return >> images.qemu_img_info(path).disk_size >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File >> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in >> qemu_img_info >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager raise >> exception.InvalidDiskInfo(reason=msg) >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager InvalidDiskInfo: >> Disk info file is invalid: qemu-img failed to execute on >> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while >> running command. >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Command: >> /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- >> env LC_ALL=C LANG=C qemu-img info >> /home/caslab/xenguest1/domains/guest1/disk.img >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Exit code: 1 >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stdout: u'' >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stderr: >> u"qemu-img: Could not open >> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open >> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n" >> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager > > > Thanks! > > > I found a crucial clue. When I tried to create an instance, every time, I got the error messages repeated continuously. It is from the log file of node2, /var/log/nova/nova-compute.log: 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager [req-f93b17df-6718-457a-9ce0-916b4e051619 cc22ec575cb44e53aced9ddf58d9e8d7 965ff1c2002d4c278b5f838dbdbbb780 - default default] Instance failed network setup after 1 attempt(s): BadRequest: Invalid input for field 'identity/password/user/password': None is not of type 'string' Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']: {'type': 'string'} On instance['identity']['password']['user']['password']: None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager Traceback (most recent call last): 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1399, in _allocate_network_async 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager bind_host_id=bind_host_id) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager bind_host_id, available_macs, requested_ports_dict) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager vif.destroy() 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager self.force_reraise() 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1057, in _update_ports_for_instance 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager port_client, instance, port_id, port_req_body) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 468, in _update_port 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager port_response = port_client.update_port(port_id, port_req_body) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret = obj(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 799, in update_port 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager revision_number=revision_number) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret = obj(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self.put(path, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret = obj(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in put 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager headers=headers, params=params) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret = obj(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager headers=headers, params=params) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager ret = obj(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager headers=headers) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 343, in do_request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self.request(url, method, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 331, in request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager resp = super(SessionClient, self).request(*args, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return auth.get_headers(self, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager token = self.get_token(session) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self.get_access(session).auth_token 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self._plugin.get_auth_ref(session, **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 177, in get_auth_ref 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager authenticated=False, log=False, **rkwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager return self.request(url, 'POST', **kwargs) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager raise exceptions.from_response(resp, method, url) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager BadRequest: Invalid input for field 'identity/password/user/password': None is not of type 'string' 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']: 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager {'type': 'string'} 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager On instance['identity']['password']['user']['password']: 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc) 2018-11-03 21:44:24.231 16305 ERROR nova.compute.manager 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [req-f93b17df-6718-457a-9ce0-916b4e051619 cc22ec575cb44e53aced9ddf58d9e8d7 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Instance failed to spawn: BadRequest: Invalid input for field 'identity/password/user/password': None is not of type 'string' Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']: {'type': 'string'} On instance['identity']['password']['user']['password']: None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Traceback (most recent call last): 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2251, in _build_resources 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] yield resources 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2031, in _build_and_run_instance 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] block_device_info=block_device_info) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3084, in spawn 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] mdevs=mdevs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5368, in _get_guest_xml 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] network_info_str = str(network_info) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 568, in __str__ 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self._sync_wrapper(fn, *args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 551, in _sync_wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.wait() 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 583, in wait 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self[:] = self._gt.wait() 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self._exit_event.wait() 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] current.throw(*self._exc) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] result = function(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 906, in context_wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return func(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1416, in _allocate_network_async 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] six.reraise(*exc_info) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1399, in _allocate_network_async 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] bind_host_id=bind_host_id) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] bind_host_id, available_macs, requested_ports_dict) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] vif.destroy() 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.force_reraise() 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] six.reraise(self.type_, self.value, self.tb) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1057, in _update_ports_for_instance 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] port_client, instance, port_id, port_req_body) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 468, in _update_port 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] port_response = port_client.update_port(port_id, port_req_body) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 799, in update_port 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] revision_number=revision_number) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.put(path, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in put 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] headers=headers, params=params) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] ret = obj(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] headers=headers) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 343, in do_request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.request(url, method, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 331, in request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] resp = super(SessionClient, self).request(*args, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.session.request(url, method, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] auth_headers = self.get_auth_headers(auth) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return auth.get_headers(self, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] token = self.get_token(session) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.get_access(session).auth_token 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] self.auth_ref = self.get_auth_ref(session) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self._plugin.get_auth_ref(session, **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 177, in get_auth_ref 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] authenticated=False, log=False, **rkwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] return self.request(url, 'POST', **kwargs) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] raise exceptions.from_response(resp, method, url) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] BadRequest: Invalid input for field 'identity/password/user/password': None is not of type 'string' 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']: 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] {'type': 'string'} 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] On instance['identity']['password']['user']['password']: 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] None (HTTP 400) (Request-ID: req-c9fac551-ebb4-4b2c-af67-5bad51d403fc) 2018-11-03 21:44:24.983 16305 ERROR nova.compute.manager [instance: 2ef5e55c-4868-4ff1-bbc9-3d82c4f99762] Above logs all occurred with only one instance creation. Although I already searched the internet, I found nothing meaningful. Just, I'm guessing that when I requested an instance creation on the controller node, it delivered the request with an empty parameter, which was deemed to None type. I mean, the empty one is password. However, I'm confused because I have set all the config files which the official documentation guided to add. As I think, the main issue is the reason the instance creation request was sent to 'node2' without password parameter. Have you ever seen the error message I had?
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack