couldn't reproduce. Also I suspect that volume attach/detach led to db
inconsistencies
** Changed in: nova
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1426524
Title:
race condition prevents intance deletion
Status in OpenStack Compute (Nova):
Invalid
Bug description:
Version: icehouse. Though looking in to the code in the master I believe bug
is still there
Hypervisor: libvirt
Frequency: very rare, under heavy load (stress tests)
Steps to reproduce: as an operator I issue "nova delete" command. Instead of
being deleted that vm gets into ERROR state.
relevant nova-compute.log: http://paste.openstack.org/show/183111/
Here's probably why it happens:
It's a race condition. There are two threads (coroutines if eventlet patched)
- thread-1 which handles termination request
(nova.compute.manager.ComputeManager.terminate_instance) and thread-2 which
dispatches events from hypervisor.
1) thread-1: manager clears (deletes) all queued events for that vm and
switches to thread-2
https://github.com/openstack/nova/blob/983f755562cb87a0b498af5d62be9bd2010bc999/nova/compute/manager.py#L2526
2) thread-2: hypervisor emits one more event and switches to thread-1 without
dispatching event
3) thread-1: manager deletes image files, marks instance as deleted in the
db. Thread finishes and exits normally
4) thread-2: manager tries to dispatch one more event. But fails as there is
no such instance anymore. To be more precise - there is no InstanceInfoCache
for that vm.
UPD: more logs:
https://www.dropbox.com/sh/r0ek3w7g95qoetw/AADTfgN9tD2Mt_fOXjB9OCzva?dl=0
- Cluster is in HA mode (three nova-api files)
- debug=False
- conductor-logs contain only "connecting, reconnecting to rabbitmq".
- cat ... | grep -A 15 -B 15 -r 09ac0ed2-07cb-4394-b0c8-aff3ab74dcdb
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1426524/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
