[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1971663
Title:
nova-compute process stuck without any actions
Status in OpenStack Compute (nova):
Expired
Bug description:
Description
===========
nova-compute process gets stuck after running for a while
Steps to reproduce
==================
Sorry, I haven't specified how to reproduce，but I know this problem has
occurred at least three times in the arm environment, but not on x86.
Expected result
===============
nova-compute run normally
Actual result
=============
2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"53988d8e-75bc-40de-b9af-ce7509f94918" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"48577b65-28e2-4858-abed-db00f90c0dc6" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock
"96408c82-5951-46e6-b105-bcb21118ed99" acquired by
"nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
:: waited 0.000s inner
/usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A
recoverable connection/channel error occurred, trying to reconnect: [Errno 104]
Connection reset by peer
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
476, in fire_timers
timer()
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line
59, in __call__
cb(*args, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line
152, in _do_acquire
waiter.switch()
greenlet.error: cannot switch to a different thread
Environment
===========
Python 3.6.8
eventlet 0.30.2
oslo.concurrency 4.4.0
oslo.messaging 12.7.1
nova-compute Stack list
======================
Stack for thread 281471352828416
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line
77, in wait
time.sleep(seconds)
Stack for thread 281473394471424
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line
77, in wait
time.sleep(seconds)
Stack for thread 281471310557696
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line
77, in wait
time.sleep(seconds)
Stack for thread 281471838974464
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line
77, in wait
time.sleep(seconds)
Stack for thread 281471847428608
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line
365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line
77, in wait
time.sleep(seconds)
Stack for thread 281471855882752
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471864336896
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471872791040
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471881245184
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471889699328
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472375845376
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472384299520
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472392753664
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472401207808
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472409661952
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472418116096
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472426570240
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472979825152
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472988279296
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472996733440
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473005187584
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473013641728
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473022095872
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473030550016
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473111552512
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75,
in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473120006656
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py",
line 205, in _native_thread
libvirt.virEventRunDefaultImpl()
File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in
virEventRunDefaultImpl
ret = libvirtmod.virEventRunDefaultImpl()
Stack for thread 281473614563696
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py",
line 42, in __thread_body
func(*args, **kwargs)
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py",
line 63, in wrap_bootstrap_inner
bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "<string>", line 167, in run
File "/usr/lib64/python3.6/code.py", line 233, in interact
more = self.push(line)
File "/usr/lib64/python3.6/code.py", line 259, in push
more = self.runsource(source, self.filename)
File "/usr/lib64/python3.6/code.py", line 75, in runsource
self.runcode(code)
File "/usr/lib64/python3.6/code.py", line 91, in runcode
exec(code, self.locals)
File "<console>", line 3, in <module>
========================================================
I tried to analyze why I got stuck, but it took me a long time to
figure it out, so looking for community help, thanks.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1971663/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
