** Changed in: ceilometer
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1075320
Title:
Instance deletion hangs after _do_terminate_instance
Status in Ceilometer:
Invalid
Status in OpenStack Compute (Nova):
Invalid
Bug description:
So when I have the below enabled and I do terminate in horizon on the
given VM it fails.
However if I restart nova-compute after commenting out the given
options it works fine.
Snippet from the Compute logs that the VM is running on.
This happens when I have:
71 instance_usage_audit = True
72 instance_usage_audit_period = hour
73 notification_driver=nova.openstack.common.notifier.rabbit_notifier
74 notification_driver=ceilometer.compute.nova_notifier
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] received
{u'_context_roles': [u'Member', u'admin'], u'_context_request_id':
u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', u'_context_quota_class': None,
u'_context_project_name': u'Smart', u'_context_service_cat
alog': [{u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'internalURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'7c348a9ac4744cedad5d13c6dd2617a5', u'publicURL': u'http
://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05'}],
u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}, {u'endpoints':
[{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne',
u'internalURL': u'http://quantum.os.lan:9696/', u'i
d': u'02bbf0add27040978d591f53c3ffedf0', u'publicURL':
u'http://quantum.bouvetlab.local:9696/'}], u'endpoints_links': [], u'type':
u'network', u'name': u'quantum'}, {u'endpoints': [{u'adminURL':
u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'internalURL': u'
http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f',
u'publicURL': u'http://glance.bouvetlab.local:9292/v2'}], u'endpoints_links':
[], u'type': u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL':
u'http://192.168.6.12:8777', u'region': u're
gionOne', u'internalURL': u'http://192.168.6.12:8777', u'id':
u'c72653c1f8044a15874eb27a5fae580e', u'publicURL':
u'http://192.168.5.52:8777'}], u'endpoints_links': [], u'type': u'metering',
u'name': u'ceilometer'}, {u'endpoints': [{u'adminURL':
u'http://heat.os.lan:8000/
v1', u'region': u'RegionOne', u'internalURL': u'http://heat.os.lan:8000/v1',
u'id': u'bb5392c96c274710bdcd8646e2510c8b', u'publicURL':
u'http://heat.os.lan:8000/v1'}], u'endpoints_links': [], u'type':
u'cloudformation', u'name': u'heat'}, {u'endpoints': [{u'adminURL': u'
http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'internalURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'e83311fd966c440384580df55cc5b5c4', u'publicURL':
u'http://cinder.bouvetlab.local:8776/v1/c9
7027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type':
u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne',
u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d
9e4ec9d41cb8a2ea8b439cdd2ee', u'publicURL':
u'http://nova.bouvetlab.local:8773/services/Cloud'}], u'endpoints_links': [],
u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL':
u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'internalURL':
u'http://sw
ift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id':
u'5f166002b1044286bc415b4fc89fdda8', u'publicURL':
u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05'}],
u'endpoints_links': [], u'type': u'object-store', u'name': u'swift'}, {u'
endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region':
u'RegionOne', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id':
u'788739bcbd06494785db65613ae2a289', u'publicURL':
u'http://keystone.bouvetlab.local:5000/v2.0'}], u'endpoints_links': []
, u'type': u'identity', u'name': u'keystone'}], u'_context_user_name':
u'endre', u'_context_auth_token': '<SANITIZED>', u'args': {u'instance':
{u'vm_state': u'active', u'availability_zone': None, u'terminated_at': None,
u'ephemeral_gb': 20, u'instance_type_id': 5, u'user
_data': None, u'vm_mode': None, u'deleted_at': None, u'reservation_id':
u'r-pb0qj603', u'id': 335, u'security_groups': [{u'deleted_at': None,
u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'name': u'default',
u'deleted': False, u'created_at': u'2012-10-11T10:00:59.000
000', u'updated_at': None, u'rules': [{u'from_port': -1, u'protocol':
u'icmp', u'group_id': None, u'deleted': False, u'created_at':
u'2012-10-12T22:49:50.000000', u'updated_at': None, u'to_port': -1,
u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'i
d': 15}, {u'from_port': 22, u'protocol': u'tcp', u'group_id': None,
u'deleted': False, u'created_at': u'2012-10-15T10:32:51.000000', u'updated_at':
None, u'to_port': 22, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0',
u'deleted_at': None, u'id': 16}, {u'from_port': 3389, u'protocol': u'tcp',
u'group_id': None, u'deleted': False, u'created_at':
u'2012-10-18T12:05:22.000000', u'updated_at': None, u'to_port': 3389,
u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 17},
{u'from_port': 1099, u'protocol': u'tcp', u'group_id': None, u'deleted': False,
u'created_at': u'2012-10-23T07:48:31.000000', u'updated_at': None, u'to_port':
1099, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None,
u'id': 18}, {u'from_port': 1, u'protocol': u'tcp', u'group_id': None,
u'deleted': False, u'created_at': u'2012-10-23T10:59:25.000000', u'updated_at':
None, u'to_port': 65535, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0',
u'deleted_at': None, u'id': 19}], u'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'id': 53, u'description': u'default'}],
u'disable_terminate': False, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78',
u'uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'server_name': None,
u'default_swap_device': None, u'info_cache': {u'instance_uuid':
u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'deleted': True, u'created_at':
u'2012-11-05T20:42:22.000000', u'updated_at': u'2012-11-05T20:57:55.000000',
u'network_info': u'[{"network": {"bridge": "", "subnets": [{"ips": [{"meta":
{}, "version": 4, "type": "fixed", "floating_ips": [], "address":
"172.16.60.3"}], "version": 4, "meta": {"dhcp_server": "172.16.60.2"}, "dns":
[], "routes": [], "cidr": "172.16.60.0/24", "gateway": {"meta": {}, "version":
4, "type": "gateway", "address": "172.16.60.1"}}], "meta": {"injected": false,
"tenant_id": "c97027dd880d4c129ae7a4ba7edade05"}, "id":
"8375aea7-bd6e-4a09-9e77-acb6494a1bbf", "label": "smart-net"}, "meta": {},
"id": "6888bc45-f8fb-461b-bb94-8e5633ab3239", "address":
"fa:16:3e:1f:a0:63"}]', u'deleted_at': u'2012-11-05T20:57:55.000000', u'id':
335}, u'hostname': u'test', u'launched_on': u'cmp01', u'display_description':
u'test', u'key_data': u'ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3c+DgMce8RjzD+2DtC2L8h/4VpmGKURKl6UlVmjiMGpZ/y7/hdqfu165xn5hxcQYmpQqnYUeUkuETTWViVJtZGHmkroM7pjVY9Uaaztndk1YsNduNs4OmIV5wGgjTQKMQNqOU21bygzcxwFNOzM0tH1LoDhco+daSbGFdhtzjENvPKBF3234sobJFPfuuDgJ1GSY16+UrYYwVmvRaK1SQUffFQT6jGTnb7Az7n/6siugy18nULKI5dNB7XGDSnKSWT8WftnzuuQShukaUzZM9tH3BLU8aqc+3112jWP6XctDKOGed2QpemifcQFchzVj9srOUqsq4fVugrAWVqSU/
client@admin', u'kernel_id': u'', u'power_state': 1,
u'default_ephemeral_device': u'/dev/vdb', u'progress': 0, u'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'launched_at':
u'2012-11-05T20:43:22.000000', u'scheduled_at': u'2012-11-05T20:42:22.000000',
u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted':
False, u'key_name': u'default', u'updated_at': u'2012-11-05T20:57:55.000000',
u'host': u'cmp01', u'display_name': u'test', u'task_state': u'deleting',
u'shutdown_terminate': False, u'architecture': None, u'root_gb': 10, u'locked':
False, u'name': u'vm-0000014f', u'created_at': u'2012-11-05T20:42:22.000000',
u'launch_index': 0, u'metadata': [], u'memory_mb': 2048, u'instance_type':
{u'disabled': False, u'root_gb': 10, u'deleted_at': None, u'name': u'm1.small',
u'deleted': False, u'created_at': None, u'ephemeral_gb': 20, u'updated_at':
None, u'memory_mb': 2048, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0,
u'is_public': True, u'flavorid': u'2', u'vcpu_weight': None, u'id': 5},
u'vcpus': 1, u'image_ref': u'0906f07d-f419-4a9d-8404-047db4717497',
u'root_device_name': u'/dev/vda', u'auto_disk_config': None, u'os_type': None,
u'config_drive': u''}}, u'_context_instance_lock_checked': False,
u'_context_is_admin': False, u'version': u'2.0', u'_context_project_id':
u'c97027dd880d4c129ae7a4ba7edade05', u'_context_timestamp':
u'2012-11-05T21:01:48.753541', u'_context_read_deleted': u'no',
u'_context_user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'method':
u'terminate_instance', u'_context_remote_address': u'192.168.5.51'} from
(pid=33107) _safe_log
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] unpacked
context: {'project_name': u'Smart', 'user_id':
u'4ce5c085e09a478ea4edcd667a92df78', 'roles': [u'Member', u'admin'],
'timestamp': u'2012-11-05T21:01:48.753541', 'auth_token': '<SANITIZED>',
'remote_address': u'192.168.5.51', 'quota_class': None, 'is_admin': False,
'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'publicURL':
u'http://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05',
u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05',
u'id': u'7c348a9ac4744cedad5d13c6dd2617a5'}], u'type': u'compute', u'name':
u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'publicURL':
u'http://quantum.bouvetlab.local:9696/', u'internalURL':
u'http://quantum.os.lan:9696/', u'id': u'02bbf0add27040978d591f53c3ffedf0'}],
u'type': u'network', u'name': u'quantum'}, {u'endpoints_links': [],
u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region':
u'RegionOne', u'publicURL': u'http://glance.bouvetlab.local:9292/v2',
u'internalURL': u'http://glance.os.lan:9292/v2', u'id':
u'c913e43a31744d499097234be00d5e6f'}], u'type': u'image', u'name': u'glance'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://192.168.6.12:8777', u'region': u'regionOne', u'publicURL':
u'http://192.168.5.52:8777', u'internalURL': u'http://192.168.6.12:8777',
u'id': u'c72653c1f8044a15874eb27a5fae580e'}], u'type': u'metering', u'name':
u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://heat.os.lan:8000/v1', u'region': u'RegionOne', u'publicURL':
u'http://heat.os.lan:8000/v1', u'internalURL': u'http://heat.os.lan:8000/v1',
u'id': u'bb5392c96c274710bdcd8646e2510c8b'}], u'type': u'cloudformation',
u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region':
u'RegionOne', u'publicURL':
u'http://cinder.bouvetlab.local:8776/v1/c97027dd880d4c129ae7a4ba7edade05',
u'internalURL':
u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id':
u'e83311fd966c440384580df55cc5b5c4'}], u'type': u'volume', u'name': u'cinder'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne',
u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud',
u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id':
u'46d5d9e4ec9d41cb8a2ea8b439cdd2ee'}], u'type': u'ec2', u'name': u'ec2'},
{u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'publicURL':
u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05',
u'internalURL':
u'http://swift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id':
u'5f166002b1044286bc415b4fc89fdda8'}], u'type': u'object-store', u'name':
u'swift'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL':
u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'publicURL':
u'http://keystone.bouvetlab.local:5000/v2.0', u'internalURL':
u'http://keystone.os.lan:5000/v2.0', u'id':
u'788739bcbd06494785db65613ae2a289'}], u'type': u'identity', u'name':
u'keystone'}], 'request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e',
'instance_lock_checked': False, 'project_id':
u'c97027dd880d4c129ae7a4ba7edade05', 'user_name': u'endre', 'read_deleted':
u'no'} from (pid=33107) _safe_log
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-05 22:02:04 DEBUG nova.utils
[req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78
c97027dd880d4c129ae7a4ba7edade05] Got semaphore
"8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68" for method "do_terminate_instance"...
from (pid=33107) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Sending
compute.instance.delete.start on notifications.info from (pid=33107) notify
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:410
2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Pool creating
new connection from (pid=33107) create
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:57
2012-11-05 22:02:04 INFO nova.openstack.common.rpc.common [-] Connected to
AMQP server on nova.os.lan:5672
2012-11-05 22:02:04 DEBUG nova.ceilometer.notifier [-] polling final stats
for u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68' from (pid=33107) notify
/opt/ceilometer/ceilometer/compute/nova_notifier.py:62
2012-11-05 22:02:04 DEBUG nova.virt.libvirt.driver
[req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78
c97027dd880d4c129ae7a4ba7edade05] Connecting to libvirt: qemu:///system from
(pid=33107) _get_connection
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:340
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._publish_service_capabilities from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._poll_rescued_instances from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Skipping
ComputeManager._sync_power_states, 10 ticks left until next run from
(pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:167
2012-11-05 22:02:48 DEBUG nova.manager
[req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task
ComputeManager._poll_bandwidth_usage from (pid=33107) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:172
To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1075320/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
