[Yahoo-eng-team] [Bug 1845759] [NEW] Openstack nova-compute service gives cant start new thread with oslo messaging

Ankit Sambhare Fri, 27 Sep 2019 21:31:29 -0700

Public bug reported:

I already have a ocata openstack version installed on ubuntu OS and some
of my compute nodes are giving "cant start new thread erros with olso
messaging messaging" and my nova-compute services are killed but i cant
find any info in syslog for the same.


ERROR logs from Compute node :

2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service [-] Error starting 
thread.
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service Traceback (most 
recent call last):
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in 
run_service
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     service.start()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.manager.init_host()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in 
init_host
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.driver.init_host(host=self.host)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in 
init_host
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self._do_quality_warnings()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in 
_do_quality_warnings
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     caps = 
self._host.get_capabilities()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in 
get_capabilities
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     xmlstr = 
self.get_connection().getCapabilities()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in 
get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     conn = 
self._get_connection()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in 
_get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     {'msg': ex})
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.force_reraise()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in 
force_reraise
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
six.reraise(self.type_, self.value, self.tb)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in 
_get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self._wrapped_conn = self._get_new_connection()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in 
_get_new_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     wrapped_conn = 
self._connect(self._uri, self._read_only)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in 
_connect
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
libvirt.openAuth, uri, auth, flags)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     rv = execute(f, 
*args, **kwargs)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     setup()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     t.start()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/threading.py", line 736, in start
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
_start_new_thread(self.__bootstrap, ())
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service error: can't start 
new thread
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service [-] Error starting 
thread.
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service Traceback (most 
recent call last):
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in 
run_service
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     service.start()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.manager.init_host()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in 
init_host
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.driver.init_host(host=self.host)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in 
init_host
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self._do_quality_warnings()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in 
_do_quality_warnings
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     caps = 
self._host.get_capabilities()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in 
get_capabilities
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     xmlstr = 
self.get_connection().getCapabilities()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in 
get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     conn = 
self._get_connection()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in 
_get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     {'msg': ex})
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.force_reraise()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in 
force_reraise
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
six.reraise(self.type_, self.value, self.tb)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in 
_get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self._wrapped_conn = self._get_new_connection()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in 
_get_new_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     wrapped_conn = 
self._connect(self._uri, self._read_only)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in 
_connect
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
libvirt.openAuth, uri, auth, flags)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     rv = execute(f, 
*args, **kwargs)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     setup()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     t.start()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/threading.py", line 736, in start
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
_start_new_thread(self.__bootstrap, ())
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service error: can't start 
new thread
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "openstack nova logs and configs file"
   
https://bugs.launchpad.net/bugs/1845759/+attachment/5292085/+files/sosreport-cmp120-20190928094143.tar.xz

** Summary changed:

- Openstack nova-compute service gives cant start ew thread with oslo messaging 
+ Openstack nova-compute service gives cant start new thread with oslo messaging

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1845759

Title:
  Openstack nova-compute service gives cant start new thread with oslo
  messaging

Status in OpenStack Compute (nova):
  New

Bug description:
  I already have a ocata openstack version installed on ubuntu OS and
  some of my compute nodes are giving "cant start new thread erros with
  olso messaging messaging" and my nova-compute services are killed but
  i cant find any info in syslog for the same.

  ERROR logs from Compute node :

  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service [-] Error 
starting thread.
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service Traceback (most 
recent call last):
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in 
run_service
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
service.start()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.manager.init_host()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in 
init_host
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.driver.init_host(host=self.host)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in 
init_host
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self._do_quality_warnings()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in 
_do_quality_warnings
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     caps = 
self._host.get_capabilities()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in 
get_capabilities
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     xmlstr = 
self.get_connection().getCapabilities()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in 
get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     conn = 
self._get_connection()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in 
_get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     {'msg': ex})
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self.force_reraise()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in 
force_reraise
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
six.reraise(self.type_, self.value, self.tb)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in 
_get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
self._wrapped_conn = self._get_new_connection()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in 
_get_new_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     wrapped_conn 
= self._connect(self._uri, self._read_only)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in 
_connect
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
libvirt.openAuth, uri, auth, flags)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     rv = 
execute(f, *args, **kwargs)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     setup()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     t.start()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File 
"/usr/lib/python2.7/threading.py", line 736, in start
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     
_start_new_thread(self.__bootstrap, ())
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service error: can't 
start new thread
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service [-] Error 
starting thread.
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service Traceback (most 
recent call last):
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in 
run_service
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
service.start()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.manager.init_host()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in 
init_host
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.driver.init_host(host=self.host)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in 
init_host
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self._do_quality_warnings()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in 
_do_quality_warnings
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     caps = 
self._host.get_capabilities()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in 
get_capabilities
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     xmlstr = 
self.get_connection().getCapabilities()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in 
get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     conn = 
self._get_connection()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in 
_get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     {'msg': ex})
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self.force_reraise()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in 
force_reraise
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
six.reraise(self.type_, self.value, self.tb)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in 
_get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
self._wrapped_conn = self._get_new_connection()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in 
_get_new_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     wrapped_conn 
= self._connect(self._uri, self._read_only)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in 
_connect
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
libvirt.openAuth, uri, auth, flags)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     rv = 
execute(f, *args, **kwargs)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     setup()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     t.start()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File 
"/usr/lib/python2.7/threading.py", line 736, in start
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     
_start_new_thread(self.__bootstrap, ())
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service error: can't 
start new thread
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1845759/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1845759] [NEW] Openstack nova-compute service gives cant start new thread with oslo messaging

Reply via email to