Public bug reported:
I'm testing openstack on my home lab (3x Dell T410 loaded). I've been
using lately openstack-ansible to deploy. Besides some issues here and
there, the overall process succeed. I'm presently hitting an issue that
I can't seem to find the possible cause.
Exhausted all hosts available for retrying ... I think this is a general
response always giving even if the issue is with the network, storage,
resource, etc.
FYI, I'm using ceph as a backend and it works good.
I'm able to create volumes, images, volumes-from-images, all from openstack
command line and horizon.
I have networks which I can ping from outside openstack.
All openstack services shows ok. From the logs, I seem to have some issues
with gnocchi, I don't know to what extend this coud prevents me from spawning
instances.
gnocchi.incoming.SackDetectionError: [Errno 2] No such file or
directory: '/var/lib/gnocchi/tmp/gnocchi-config'
I have full logs, but I can't seem to find the culprit.
In the compute-node, I have this..pretty cryptic...:
ERROR nova.compute.manager [req-66f25425-c735-44dd-aa69-d8e1c904f17e
0540dc5ba00641c1b86a269b99d4b323 ddc8fdea71a943a7b36c4ae66b2f44ed - default
default] [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Failed to build and
run instance: libvirt.libvirtError: Failed to activate service
'org.freedesktop.machine1': timed out
(service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
Traceback (most recent call last):#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/compute/manager.py",
line 2384, in _build_and_run_instance#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.driver.spawn(context, instance, image_meta,#0122021-02-09 09:41:39.705
408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 3682, in spawn#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self._create_guest_with_network(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6683, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self._cleanup_failed_start(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6652, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
guest = self._create_guest(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6585, in _create_guest#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
guest.launch(pause=pause)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py",
line 158, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
LOG.exception('Error launching a defined domain with XML: %s',#0122021-02-09
09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py",
line 155, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
return self._domain.createWithFlags(flags)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 190, in doit#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
result = proxy_call(self._autowrap, f, *args, **kwargs)#0122021-02-09
09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 148, in proxy_call#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
rv = execute(f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(c, e, tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
rv = meth(*args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/libvirt.py",
line 1265, in createWithFlags#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1':
timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915364
Title:
fail to launch instances
Status in OpenStack Compute (nova):
New
Bug description:
I'm testing openstack on my home lab (3x Dell T410 loaded). I've been
using lately openstack-ansible to deploy. Besides some issues here
and there, the overall process succeed. I'm presently hitting an
issue that I can't seem to find the possible cause.
Exhausted all hosts available for retrying ... I think this is a
general response always giving even if the issue is with the network,
storage, resource, etc.
FYI, I'm using ceph as a backend and it works good.
I'm able to create volumes, images, volumes-from-images, all from openstack
command line and horizon.
I have networks which I can ping from outside openstack.
All openstack services shows ok. From the logs, I seem to have some issues
with gnocchi, I don't know to what extend this coud prevents me from spawning
instances.
gnocchi.incoming.SackDetectionError: [Errno 2] No such file or
directory: '/var/lib/gnocchi/tmp/gnocchi-config'
I have full logs, but I can't seem to find the culprit.
In the compute-node, I have this..pretty cryptic...:
ERROR nova.compute.manager [req-66f25425-c735-44dd-aa69-d8e1c904f17e
0540dc5ba00641c1b86a269b99d4b323 ddc8fdea71a943a7b36c4ae66b2f44ed - default
default] [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Failed to build and
run instance: libvirt.libvirtError: Failed to activate service
'org.freedesktop.machine1': timed out
(service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
Traceback (most recent call last):#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/compute/manager.py",
line 2384, in _build_and_run_instance#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.driver.spawn(context, instance, image_meta,#0122021-02-09 09:41:39.705
408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 3682, in spawn#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self._create_guest_with_network(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6683, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self._cleanup_failed_start(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6652, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
guest = self._create_guest(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 6585, in _create_guest#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
guest.launch(pause=pause)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py",
line 158, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
LOG.exception('Error launching a defined domain with XML: %s',#0122021-02-09
09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py",
line 155, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
return self._domain.createWithFlags(flags)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 190, in doit#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
result = proxy_call(self._autowrap, f, *args, **kwargs)#0122021-02-09
09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 148, in proxy_call#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
rv = execute(f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
six.reraise(c, e, tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py",
line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
rv = meth(*args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
File
"/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/libvirt.py",
line 1265, in createWithFlags#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]
libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1':
timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915364/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
