This bug lacks the necessary information to effectively reproduce and
fix it, therefore it has been closed. Feel free to reopen the bug by
providing the requested information and set the bug status back to
''New''.
** Changed in: nova
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1075320
Title:
Instance deletion hangs after _do_terminate_instance
Status in Ceilometer:
Confirmed
Status in OpenStack Compute (Nova):
Invalid
Bug description:
So when I have the below enabled and I do terminate in horizon on the
given VM it fails.
However if I restart nova-compute after commenting out the given
options it works fine.
Snippet from the Compute logs that the VM is running on.
This happens when I have:
71 instance_usage_audit = True
72 instance_usage_audit_period = hour
73 notification_driver=nova.openstack.common.notifier.rabbit_notifier
74 notification_driver=ceilometer.compute.nova_notifier
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] received
{u'_context_roles': [u'Member', u'admin'], u'_context_request_id':
u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', u'_context_quota_class': None,
u'_context_project_name': u'Smart', u'_context_service_cat
alog': [{u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'internalURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'7c348a9ac4744cedad5d13c6dd2617a5', u'publicURL': u'http
://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05'}],
u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}, {u'endpoints':
[{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne',
u'internalURL': u'http://quantum.os.lan:9696/', u'i
d': u'02bbf0add27040978d591f53c3ffedf0', u'publicURL':
u'http://quantum.bouvetlab.local:9696/'}], u'endpoints_links': [], u'type':
u'network', u'name': u'quantum'}, {u'endpoints': [{u'adminURL':
u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'internalURL': u'
http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f',
u'publicURL': u'http://glance.bouvetlab.local:9292/v2'}], u'endpoints_links':
[], u'type': u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL':
u'http://192.168.6.12:8777', u'region': u're
gionOne', u'internalURL': u'http://192.168.6.12:8777', u'id':
u'c72653c1f8044a15874eb27a5fae580e', u'publicURL':
u'http://192.168.5.52:8777'}], u'endpoints_links': [], u'type': u'metering',
u'name': u'ceilometer'}, {u'endpoints': [{u'adminURL':
u'http://heat.os.lan:8000/
v1', u'region': u'RegionOne', u'internalURL': u'http://heat.os.lan:8000/v1',
u'id': u'bb5392c96c274710bdcd8646e2510c8b', u'publicURL':
u'http://heat.os.lan:8000/v1'}], u'endpoints_links': [], u'type':
u'cloudformation', u'name': u'heat'}, {u'endpoints': [{u'adminURL': u'
http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'internalURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'e83311fd966c440384580df55cc5b5c4', u'publicURL':
u'http://cinder.bouvetlab.local:8776/v1/c9
7027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type':
u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne',
u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d
9e4ec9d41cb8a2ea8b439cdd2ee', u'publicURL':
u'http://nova.bouvetlab.local:8773/services/Cloud'}], u'endpoints_links': [],
u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL':
u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'internalURL':
u'http://sw
ift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id':
u'5f166002b1044286bc415b4fc89fdda8', u'publicURL':
u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05'}],
u'endpoints_links': [], u'type': u'object-store', u'name': u'swift'}, {u'
endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region':
u'RegionOne', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id':
u'788739bcbd06494785db65613ae2a289', u'publicURL':
u'http://keystone.bouvetlab.local:5000/v2.0'}], u'endpoints_links': []
, u'type': u'identity', u'name': u'keystone'}], u'_context_user_name':
u'endre', u'_context_auth_token': '<SANITIZED>', u'args': {u'instance':
{u'vm_state': u'active', u'availability_zone': None, u'terminated_at': None,
u'ephemeral_gb': 20, u'instance_type_id': 5, u'user
_data': None, u'vm_mode': None, u'deleted_at': None, u'reservation_id':
u'r-pb0qj603', u'id': 335, u'security_groups': [{u'deleted_at': None,
u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'name': u'default',
u'deleted': False, u'created_at': u'2012-10-11T10:00:59.000
000', u'updated_at': None, u'rules': [{u'from_port': -1, u'protocol':
u'icmp', u'group_id': None, u'deleted': False, u'created_at':
u'2012-10-12T22:49:50.000000', u'updated_at': None, u'to_port': -1,
u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'i
d': 15}, {u'from_port': 22, u'protocol': u'tcp', u'group_id': None,
u'deleted': False, u'created_at': u'2012-10-15T10:32:51.000000', u'updated_at':
None, u'to_port': 22, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0',
u'deleted_at': None, u'id': 16}, {u'from_port': 3389, u'protocol': u'tcp',
u'group_id': None, u'deleted': False, u'created_at':
u'2012-10-18T12:05:22.000000', u'updated_at': None, u'to_port': 3389,
u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 17},
{u'from_port': 1099, u'protocol': u'tcp', u'group_id': None, u'deleted': False,
u'created_at': u'2012-10-23T07:48:31.000000', u'updated_at': None, u'to_port':
1099, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None,
u'id': 18}, {u'from_port': 1, u'protocol': u'tcp', u'group_id': None,
u'deleted': False, u'created_at': u'2012-10-23T10:59:25.000000', u'updated_at':
None, u'to_port': 65535, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0',
u'deleted_at': None, u'id': 19}], u'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'id': 53, u'description': u'default'}],
u'disable_terminate': False, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78',
u'uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'server_name': None,
u'default_swap_device': None, u'info_cache': {u'instance_uuid':
u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'deleted': True, u'created_at':
u'2012-11-05T20:42:22.000000', u'updated_at': u'2012-11-05T20:57:55.000000',
u'network_info': u'[{"network": {"bridge": "", "subnets": [{"ips": [{"meta":
{}, "version": 4, "type": "fixed", "floating_ips": [], "address":
"172.16.60.3"}], "version": 4, "meta": {"dhcp_server": "172.16.60.2"}, "dns":
[], "routes": [], "cidr": "172.16.60.0/24", "gateway": {"meta": {}, "version":
4, "type": "gateway", "address": "172.16.60.1"}}], "meta": {"injected": false,
"tenant_id": "c97027dd880d4c129ae7a4ba7edade05"}, "id":
"8375aea7-bd6e-4a09-9e77-acb6494a1bbf", "label": "smart-net"}, "meta": {},
"id": "6888bc45-f8fb-461b-bb94-8e5633ab3239", "address":
"fa:16:3e:1f:a0:63"}]', u'deleted_at': u'2012-11-05T20:57:55.000000', u'id':
335}, u'hostname': u'test', u'launched_on': u'cmp01', u'display_description':
u'test', u'key_data': u'ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3c+DgMce8RjzD+2DtC2L8h/4VpmGKURKl6UlVmjiMGpZ/y7/hdqfu165xn5hxcQYmpQqnYUeUkuETTWViVJtZGHmkroM7pjVY9Uaaztndk1YsNduNs4OmIV5wGgjTQKMQNqOU21bygzcxwFNOzM0tH1LoDhco+daSbGFdhtzjENvPKBF3234sobJFPfuuDgJ1GSY16+UrYYwVmvRaK1SQUffFQT6jGTnb7Az7n/6siugy18nULKI5dNB7XGDSnKSWT8WftnzuuQShukaUzZM9tH3BLU8aqc+3112jWP6XctDKOGed2QpemifcQFchzVj9srOUqsq4fVugrAWVqSU/
client@admin', u'kernel_id': u'', u'power_state': 1,
u'default_ephemeral_device': u'/dev/vdb', u'progress': 0, u'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'launched_at':
u'2012-11-05T20:43:22.000000', u'scheduled_at': u'2012-11-05T20:42:22.000000',
u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted':
False, u'key_name': u'default', u'updated_at': u'2012-11-05T20:57:55.000000',
u'host': u'cmp01', u'display_name': u'test', u'task_state': u'deleting',
u'shutdown_terminate': False, u'architecture': None, u'root_gb': 10, u'locked':
False, u'name': u'vm-0000014f', u'created_at': u'2012-11-05T20:42:22.000000',
u'launch_index': 0, u'metadata': [], u'memory_mb': 2048, u'instance_type':
{u'disabled': False, u'root_gb': 10, u'deleted_at': None, u'name': u'm1.small',
u'deleted': False, u'created_at': None, u'ephemeral_gb': 20, u'updated_at':
None, u'memory_mb': 2048, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0,
u'is_public': True, u'flavorid': u'2', u'vcpu_weight': None, u'id': 5},
u'vcpus': 1, u'image_ref': u'0906f07d-f419-4a9d-8404-047db4717497',
u'root_device_name': u'/dev/vda', u'auto_disk_config': None, u'os_type': None,
u'config_drive': u''}}, u'_context_instance_lock_checked': False,
u'_context_is_admin': False, u'version': u'2.0', u'_context_project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'_context_timestamp':
u'2012-11-05T21:01:48.753541', u'_context_read_deleted': u'no',
u'_context_user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'method':
u'terminate_instance', u'_context_remote_address': u'192.168.5.51'} from
(pid=33107) _safe_log
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] unpacked
context: {'project_name': u'Smart', 'user_id':
u'4ce5c085e09a478ea4edcd667a92df78', 'roles': [u'Member', u'admin'],
'timestamp': u'2012-11-05T21:01:48.753541', 'auth_token': '<SANITIZED>',
'remote_address': u'192.168.5.51', 'quota_class': None, 'is_admin': False,
'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'publicURL':
u'http://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05',
u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05',
u'id': u'7c348a9ac4744cedad5d13c6dd2617a5'}], u'type': u'compute', u'name':
u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'publicURL':
u'http://quantum.bouvetlab.local:9696/', u'internalURL':
u'http://quantum.os.lan:9696/', u'id': u'02bbf0add27040978d591f53c3ffedf0'}],
u'type': u'network', u'name': u'quantum'}, {u'endpoints_links': [],
u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region':
u'RegionOne', u'publicURL': u'http://glance.bouvetlab.local:9292/v2',
u'internalURL': u'http://glance.os.lan:9292/v2', u'id':
u'c913e43a31744d499097234be00d5e6f'}], u'type': u'image', u'name': u'glance'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://192.168.6.12:8777', u'region': u'regionOne', u'publicURL':
u'http://192.168.5.52:8777', u'internalURL': u'http://192.168.6.12:8777',
u'id': u'c72653c1f8044a15874eb27a5fae580e'}], u'type': u'metering', u'name':
u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://heat.os.lan:8000/v1', u'region': u'RegionOne', u'publicURL':
u'http://heat.os.lan:8000/v1', u'internalURL': u'http://heat.os.lan:8000/v1',
u'id': u'bb5392c96c274710bdcd8646e2510c8b'}], u'type': u'cloudformation',
u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'publicURL':
u'http://cinder.bouvetlab.local:8776/v1/c97027dd880d4c129ae7a4ba7edade05',
u'internalURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'e83311fd966c440384580df55cc5b5c4'}], u'type': u'volume', u'name': u'cinder'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne',
u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud',
u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id':
u'46d5d9e4ec9d41cb8a2ea8b439cdd2ee'}], u'type': u'ec2', u'name': u'ec2'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'publicURL':
u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05',
u'internalURL':
u'http://swift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id':
u'5f166002b1044286bc415b4fc89fdda8'}], u'type': u'object-store', u'name':
u'swift'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'publicURL':
u'http://keystone.bouvetlab.local:5000/v2.0', u'internalURL':
u'http://keystone.os.lan:5000/v2.0', u'id':
u'788739bcbd06494785db65613ae2a289'}], u'type': u'identity', u'name':
u'keystone'}], 'request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e',
'instance_lock_checked': False, 'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', 'user_name': u'endre', 'read_deleted':
u'no'} from (pid=33107) _safe_log
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-05 22:02:04 DEBUG nova.utils
[req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78
c97027dd880d4c129ae7a4ba7edade05] Got semaphore
"8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68" for method "do_terminate_instance"...
from (pid=33107) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Sending
compute.instance.delete.start on notifications.info from (pid=33107) notify
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:410
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Pool creating
new connection from (pid=33107) create
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:57
2012-11-05 22:02:04 INFO nova.openstack.common.rpc.common [-] Connected to
AMQP server on nova.os.lan:5672
2012-11-05 22:02:04 DEBUG nova.ceilometer.notifier [-] polling final stats
for u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68' from (pid=33107) notify
/opt/ceilometer/ceilometer/compute/nova_notifier.py:62
2012-11-05 22:02:04 DEBUG nova.virt.libvirt.driver
[req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78
c97027dd880d4c129ae7a4ba7edade05] Connecting to libvirt: qemu:///system from
(pid=33107) _get_connection
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:340
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._publish_service_capabilities from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._poll_rescued_instances from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Skipping
ComputeManager._sync_power_states, 10 ticks left until next run from
(pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:167
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._poll_bandwidth_usage from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1075320/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
