Hi Juha, Thanks for the further detail, I see the problem now and have filed a bug[1] which I will fix for icehouse3 and backport to stable/havana.
Note though that this is orthogonal to your main issue with autoscaling. Cheers, Eoghan [1] https://bugs.launchpad.net/ceilometer/+bug/1284073 ----- Original Message ----- > Hi Eoghan, > > Thank you very much for the reply. I'll try and add the "repeat_actions" to > my Heat template. > > >>>I'd be interested in hearing which actual metering messages are being > rejected with not okForStorage. > > Here is some debug logging from the time not okForStorage errors occur: > > <44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > e084351a131247dea21135e837419812. > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-fa96434e-7b27-4261-af17-73e79b70cc > 7a', u'args': {u'data': [{u'counter_name': u'disk.ephemeral.size', > u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'0b900e89-cf10-44a8-a79c-f2f68f5 > 5fc07', u'timestamp': u'2014-02-24 11:59:55.295790', u'message_signature': > u'be29bc5f477bb24c938d028b2b10c301867744031241bfbde3cda0c59ef7dbe9', > u'resource_metadata': > {u'state_description': u'', u'event_type': u'compute.instance.delete.end', > u'availability_zone': None, u'terminated_at': > u'2014-02-24T11:59:54.965388', u'ephemeral_ > gb': 0, u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id': > u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', > u'user_id': u'ef4e9832 > 91ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted', > u'launched_at': u'2014-02-24T11:58:16.000000', u'm > etadata': {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': > u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node': > u'node-17', u'ramdi > sk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, > u'kernel_id': u'', u'host': u'compute.node-17', u'display_name': > u'tyky-Group_A-wste7bd63rk > l-Group_A-1-xzgy6fkgqvhc', u'image_ref_url': u' > http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'root_gb': 1, u'tenant_id': u'efcca4ba425c4be > da73eb31a54df931a', u'created_at': u'2014-02-24 11:58:08+00:00', > u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, > u'image_meta': {u'min_disk': u'1', u'c > ontainer_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', > u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'os_type' > : None, u'instance_flavor_id': u'1'}, u'source': u'openstack', > u'counter_unit': u'GB', u'counter_volume': 0, u'project_id': > u'efcca4ba425c4beda73eb31a54df931a', u'me > ssage_id': u'2cffb528-9d4b-11e3-b0e0-080027e519cb', u'counter_type': > u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', > u'_context_show_deleted': False, u'_context_ > tenant': None, u'_unique_id': u'e084351a131247dea21135e837419812', > u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': > False, u'_context_user': No > ne, u'method': u'record_metering_data'} > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked > context: {'read_only': False, 'show_deleted': False, 'auth_token' > : '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': > u'req-fa96434e-7b27-4261-af17-73e79b70cc7a', 'tenant': None} > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1 > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1 > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering > data disk.ephemeral.size for 0b900e89-cf10-44a8-a79c-f2f68f55 > fc07 @ 2014-02-24 11:59:55.295790: 0 > <43>Feb 24 11:59:55 node-16 > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record > metering data: not okForStorage > Traceback (most recent call last): > File > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", > line 65, in record_metering_data > self.storage_conn.record_metering_data(meter) > File > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line > 417, in record_metering_data > upsert=True, > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, > in update > check_keys, self.__uuid_subtype), safe) > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > 969, in _send_message > rv = self.__check_response_to_last_error(response) > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > 911, in __check_response_to_last_error > raise OperationFailure(details["err"], details["code"]) > OperationFailure: not okForStorage > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open > <44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > 6bf474db7f9f4bc3a3a4342508c54218. > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1 > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1 > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', > u'args': {u'data': [{u'counter_name': u'disk.root.size', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'timestamp': u'2014-02-24 > 11:59:55.295790', u'message_signature': > u'49ea1df965fd09b6ce3c84812977e9c21474030006510ef402afd6b2e12c9f90', > u'resource_metadata': {u'state_description': u'', u'event_type': > u'compute.instance.delete.end', u'availability_zone': None, > u'terminated_at': u'2014-02-24T11:59:54.965388', u'ephemeral_gb': 0, > u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id': > u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', > u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted', > u'launched_at': u'2014-02-24T11:58:16.000000', u'metadata': > {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': > u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node': > u'node-17', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, > u'access_ip_v4': None, u'kernel_id': u'', u'host': u'compute.node-17', > u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-1-xzgy6fkgqvhc', > u'image_ref_url': u' > http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', > u'created_at': u'2014-02-24 11:58:08+00:00', u'memory_mb': 512, > u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': > u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': > u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'os_type': None, u'instance_flavor_id': u'1'}, > u'source': u'openstack', u'counter_unit': u'GB', u'counter_volume': 1, > u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': > u'2d03ed96-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, > u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, > u'_context_tenant': None, u'_unique_id': > u'6bf474db7f9f4bc3a3a4342508c54218', u'_context_is_admin': True, > u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, > u'method': u'record_metering_data'} > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked > context: {'read_only': False, 'show_deleted': False, 'auth_token': > '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': > u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', 'tenant': None} > <47>Feb 24 11:59:55 node-16 > <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering > data disk.root.size for 0b900e89-cf10-44a8-a79c-f2f68f55fc07 @ 2014-02-24 > 11:59:55.295790: 1 > <47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open > <43>Feb 24 11:59:55 node-16 > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record > metering data: not okForStorage > Traceback (most recent call last): > File > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", > line 65, in record_metering_data > self.storage_conn.record_metering_data(meter) > File > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line > 417, in record_metering_data > upsert=True, > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, > in update > check_keys, self.__uuid_subtype), safe) > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > 969, in _send_message > rv = self.__check_response_to_last_error(response) > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > 911, in __check_response_to_last_error > raise OperationFailure(details["err"], details["code"]) > OperationFailure: not okForStorage > <44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > 6a353b849919442ababb50bb41199f40. > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1 > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1 > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86', > u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 > 12:00:01.598433', u'message_signature': > u'a9a962db100587c8b19ec8140510e65e38e651e8b4fc977d3255ccd2a02f6477', > u'resource_metadata': {u'state_description': u'', u'event_type': > u'compute.instance.exists', u'availability_zone': None, u'terminated_at': > u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, > u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', > u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': > [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, > u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, > u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': > None, u'id': 170}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': > u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', > u'deleted_at': None, u'id': 171}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', > u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': > u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, > u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', > u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', > u'image_ref_url': u' > http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, > u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': > u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': > u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', > u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', > u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', > u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': > u'efcca4ba425c4beda73eb31a54df931a', u'message_id': > u'30adebae-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, > u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, > u'_context_tenant': None, u'_unique_id': > u'6a353b849919442ababb50bb41199f40', u'_context_is_admin': True, > u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, > u'method': u'record_metering_data'} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked > context: {'read_only': False, 'show_deleted': False, 'auth_token': > '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': > u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86', 'tenant': None} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering > data instance:m1.tiny for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24 > 12:00:01.598433: 1 > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open > <44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > 6d1b3d50a1344561abd0b18ed91c0415. > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1 > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1 > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-5543e2de-cc1d-48d0-975c-62e103d17498', > u'args': {u'data': [{u'counter_name': u'instance', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 > 12:00:01.598433', u'message_signature': > u'd8fb3729478f1930acfec6de47a786a2a68d58bedf9e9ea31f24fb107d548b26', > u'resource_metadata': {u'state_description': u'', u'event_type': > u'compute.instance.exists', u'availability_zone': None, u'terminated_at': > u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, > u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', > u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': > [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, > u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, > u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': > None, u'id': 170}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': > u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', > u'deleted_at': None, u'id': 171}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', > u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': > u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, > u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', > u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', > u'image_ref_url': u' > http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, > u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': > u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': > u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', > u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', > u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', > u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': > u'efcca4ba425c4beda73eb31a54df931a', u'message_id': > u'30af474c-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, > u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, > u'_context_tenant': None, u'_unique_id': > u'6d1b3d50a1344561abd0b18ed91c0415', u'_context_is_admin': True, > u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, > u'method': u'record_metering_data'} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked > context: {'read_only': False, 'show_deleted': False, 'auth_token': > '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': > u'req-5543e2de-cc1d-48d0-975c-62e103d17498', 'tenant': None} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering > data instance for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24 > 12:00:01.598433: 1 > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open > <44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > 0e82e0a8ce8b4a6aa0a1de775449c43b. > <44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > Publishing 1 samples on metering > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making > asynchronous cast on metering... > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is > be9f47cffe0e42c7afb18ed0a25e2c8f. > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1 > <47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1 > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16', > u'args': {u'data': [{u'counter_name': u'vcpus', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 > 12:00:01.598433', u'message_signature': > u'8bc94bd242cb9243fce3a78867b0cb4630abc69602f84e644089c1ff85d586e7', > u'resource_metadata': {u'state_description': u'', u'event_type': > u'compute.instance.exists', u'availability_zone': None, u'terminated_at': > u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, > u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', > u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': > [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, > u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, > u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': > None, u'id': 170}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': > u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', > u'deleted_at': None, u'id': 171}, {u'instance_uuid': > u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': > u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', > u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': > u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, > u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', > u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', > u'image_ref_url': u' > http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, > u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': > u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': > u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', > u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', > u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', > u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', > u'counter_unit': u'vcpu', u'counter_volume': 1, u'project_id': > u'efcca4ba425c4beda73eb31a54df931a', u'message_id': > u'30b1b3ec-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, > u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, > u'_context_tenant': None, u'_unique_id': > u'0e82e0a8ce8b4a6aa0a1de775449c43b', u'_context_is_admin': True, > u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, > u'method': u'record_metering_data'} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked > context: {'read_only': False, 'show_deleted': False, 'auth_token': > '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': > u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16', 'tenant': None} > <47>Feb 24 12:00:01 node-16 > <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received > {u'_context_request_id': u'req-fe67f75f-f9e0-495e-8afd-8cfe5bb6414d', > u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': > u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'timestamp': u'2014-02-24 > 12:00:01.630458', u'message_signature': > u'1e5462bfc6dad2a2cbb1cd00b31de14ed403608f252f4e8dbafb4b2bcb470a28', > u'resource_metadata': {u'state_description': u'', u'event_type': > u'compute.instance.exists', u'availability_zone': None, u'terminated_at': > u'2014-02-24T11:15:04.000000', u'ephemeral_gb': 0, u'instance_type_id': 2, > u'bandwidth': {}, u'deleted_at': u'2014-02-24T11:15:04.000000', > u'reservation_id': u'r-548dio0m', u'instance_id': > u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'user_id': > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': > u'tyky-group-a-yal5hnj7o4vw-group-a-0-bb43lazasuub', u'state': u'deleted', > u'launched_at': u'2014-02-24T11:14:11.000000', u'metadata': [], u'node': > u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, > u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', > u'display_name': u'tyky-Group_A-yal5hnj7o4vw-Group_A-0-bb43lazasuub', > u'image_ref_url': u' > http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b', > u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, > u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': > u'2014-02-24T11:14:03.000000', u'memory_mb': 512, u'instance_type': > u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', > u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', > u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, > u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', > u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': > u'efcca4ba425c4beda73eb31a54df931a', u'message_id': > u'30b2b86e-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, > u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, > u'_context_tenant': None, u'_unique_id': > u'be9f47cffe0e42c7afb18ed0a25e2c8f', u'_context_is_admin': True, > u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, > u'method': u'record_metering_data'} > > Br, > -Juha > > > > On 24 February 2014 11:46, Eoghan Glynn <egl...@redhat.com> wrote: > > > > > Hi Juha, > > > > Are you using Havana? > > > > If so, your alarm resource templates seem to be missing setting the > > "repeat_actions" attribute to true. Continuous notification was required > > in order for the Heat autoscaling cooldown logic (and continued scaling > > actions) to work. > > > > This was a quirk of the Heat implementation in Havan, but TBH from a > > usability PoV should have always had this set to True by default. This > > is fixed now on Heat:master: > > > > https://github.com/openstack/heat/commit/4dc987ef > > > > but hasn't been backported to Heat:stable/stable. > > > > The issue you see with the instance.scheduled message being rejected with > > "signature invalid" is not related to autoscaling, and in case I've fixed > > it in Ceilometer:master and stable/havana: > > > > https://bugs.launchpad.net/ceilometer/+bug/1262255 > > > > released in: > > > > https://launchpad.net/ceilometer/+milestone/2013.2.2 > > > > The issue you see with mongo complaining that a metering message is not > > okForStorage is generally caused by a stray period ('.') in some resource > > metadata key. > > > > Where the metadata is prefixed by 'metering.', then the periods are > > mapped to underscores, so in your case: > > > > { "Key" : "metering.server_group", "Value" : "Group_A" }, > > ^^^^^^^^^^ > > > > that mapping should have occurred. In any case, I'd be interested in > > hearing > > which actual metering messages are being rejected with not okForStorage. > > > > Thanks, > > Eoghan > > > > > Hi, > > > > > > I'm having some problems concerning auto scaling feature. > > > Any ideas? > > > > > > First scaling up and down is working just fine. But then when tested > > later on > > > scaling down/up is no longer working properly. > > > Scaling down may occur even it shouldn't or scaling up doesn't occur > > even it > > > should. When in this situation I remove all the > > > received metric data from the DB, auto scaling starts to work again. > > > > > > Ceilometer is configured to use Mongo and the auto scaling is based on > > the > > > cpu_util metrics. > > > > > > Related configurations: > > > ----------------------- > > > /etc/ceilometer/pipeline.yaml on compute nodes: > > > > > > name: cpu_pipeline > > > interval: 15 > > > > > > /etc/ceilometer/ceilometer.conf on controller: > > > evaluation_interval=15 > > > > > > Heat template used: > > > ------------------- > > > "Resources" : { > > > > > > "Group_A" : { > > > "Type" : "AWS::AutoScaling::AutoScalingGroup", > > > "Properties" : { > > > "AvailabilityZones" : { "Fn::GetAZs" : ""}, > > > "LaunchConfigurationName" : { "Ref" : "Group_A_Config" }, > > > "MinSize" : "1", > > > "MaxSize" : "3", > > > "Tags" : [ > > > { "Key" : "metering.server_group", "Value" : "Group_A" }, > > > { "Key" : "custom_metadata", "Value" : "test" } > > > ], > > > "VPCZoneIdentifier" : [ { "Ref" : "PrivateSubnetId" } ] > > > } > > > }, > > > > > > "Group_A_Config" : { > > > "Type" : "AWS::AutoScaling::LaunchConfiguration", > > > "Properties": { > > > "ImageId" : { "Ref" : "ImageId" }, > > > "InstanceType" : { "Ref" : "InstanceType" }, > > > "KeyName" : { "Ref" : "KeyName" } > > > } > > > }, > > > > > > "ScaleUpPolicy" : { > > > "Type" : "AWS::AutoScaling::ScalingPolicy", > > > "Properties" : { > > > "AdjustmentType" : "ChangeInCapacity", > > > "AutoScalingGroupName" : { "Ref" : "Group_A" }, > > > "Cooldown" : "20", > > > "ScalingAdjustment" : "1" > > > } > > > }, > > > > > > "ScaleDownPolicy" : { > > > "Type" : "AWS::AutoScaling::ScalingPolicy", > > > "Properties" : { > > > "AdjustmentType" : "ChangeInCapacity", > > > "AutoScalingGroupName" : { "Ref" : "Group_A" }, > > > "Cooldown" : "20", > > > "ScalingAdjustment" : "-1" > > > } > > > }, > > > > > > "CPUAlarmHigh": { > > > "Type": "OS::Ceilometer::Alarm", > > > "Properties": { > > > "description": "Scale-up if CPU is greater than 90% for 20 seconds", > > > "meter_name": "cpu_util", > > > "statistic": "avg", > > > "period": "20", > > > "evaluation_periods": "1", > > > "threshold": "90", > > > "alarm_actions": > > > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ], > > > "matching_metadata": > > > {"metadata.user_metadata.server_group": "Group_A" }, > > > "comparison_operator": "gt" > > > } > > > }, > > > > > > "CPUAlarmLow": { > > > "Type": "OS::Ceilometer::Alarm", > > > "Properties": { > > > "description": "Scale-down if CPU is less than 50% for 20 seconds", > > > "meter_name": "cpu_util", > > > "statistic": "avg", > > > "period": "20", > > > "evaluation_periods": "1", > > > "threshold": "50", > > > "alarm_actions": > > > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ], > > > "matching_metadata": > > > {"metadata.user_metadata.server_group": "Group_A" }, > > > "comparison_operator": "lt" > > > } > > > > > > In ceilometer logs I can see the following kind of warnings: > > > > > > <44>Feb 24 08:41:08 node-16 > > > ceilometer-ceilometer.collector.dispatcher.database WARNING: message > > > signature invalid, discarding message: {u'counter_name': > > > u'instance.scheduled', u'user_id': None, u'message_signature': > > > u'd1b49ddf004edc5b7a8dc9405b42a71f2ae975d04c25838c3dc0ea0e6f6e4edd', > > > u'timestamp': u'2014-02-24 08:41:08.334580', u'resource_id': > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'message_id': > > > u'67e611e4-9d2f-11e3-81f1-080027e519cb', u'source': u'openstack', > > > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': > > > u'efcca4ba425c4beda73eb31a54df931a', u'resource_metadata': > > {u'instance_id': > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'weighted_host': {u'host': > > > u'node-18', u'weight': 3818.0}, u'host': u'scheduler.node-16', > > > u'request_spec': {u'num_instances': 1, u'block_device_mapping': > > > [{u'instance_uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c', > > > u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True, > > > u'no_device': None, u'connection_info': None, u'volume_id': None, > > > u'device_name': None, u'disk_bus': None, u'image_id': > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'source_type': u'image', > > > u'device_type': u'disk', u'snapshot_id': None, u'destination_type': > > > u'local', u'volume_size': None}], u'image': {u'status': u'active', > > u'name': > > > u'cirrosImg', u'deleted': False, u'container_format': u'bare', > > > u'created_at': u'2014-02-12T08:46:04.000000', u'disk_format': u'qcow2', > > > u'updated_at': u'2014-02-12T08:46:04.000000', u'properties': {}, > > > u'min_disk': 0, u'min_ram': 0, u'checksum': > > > u'50bdc35edb03a38d91b1b071afb20a3c', u'owner': > > > u'efcca4ba425c4beda73eb31a54df931a', u'is_public': True, u'deleted_at': > > > None, u'id': u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'size': 9761280}, > > > u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': > > 0, > > > u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0, > > > u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2}, > > > u'instance_properties': {u'vm_state': u'building', u'availability_zone': > > > None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2, > > > u'user_data': > > u'Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PT0 > > > ... > > > , u'cleaned': False, u'vm_mode': None, u'deleted_at': None, > > > u'reservation_id': u'r-l91mh33v', u'id': 274, u'security_groups': > > > {u'objects': []}, u'disable_terminate': False, u'root_device_name': None, > > > u'display_name': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey', > > > u'uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c', u'default_swap_device': > > > None, u'info_cache': {u'instance_uuid': > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'network_info': []}, > > u'hostname': > > > u'tyky-group-a-55cklit7nvbq-group-a-2-yis32na5m7ey', u'launched_on': > > None, > > > u'display_description': > > u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey', > > > u'key_data': u'ssh-rsa > > > > > AAAAB3NzaC1yc2EAAAADAQABAAABAQC39hmz8e40Xv/+QKkLyRA7j02RfIG61cr1j41RftnkOF3ZbwBzi7qibsOA3gC9Ln05YbB6z2/iUnQzxQsoOpmlnXuv2O296utY2ZCTKhdFSzn2Ot7l635zEXkivMc97wz4bITtaBTjX3nV6sXOfevdTIOJeC11SqxmfNRRzXcz9fRv6kLjz7IrA0tvRTp2xDVtFEj+vFLWaXc3TcUSygxiSLeAuNkH1rZ9jVuHXXvzb/e7navrGyJec2P86AQg2TUk77MhLjPcbyKiJJK0DhK6zOkZUWXtgIVQx7+gO/Xs2QgQHcw+VdzRzpJK+/EOzUOU8IDWNnyfaJEnQEoX2oMj > > > Generated by Nova\n', u'deleted': False, u'config_drive': u'', > > > u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0, > > > u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'launched_at': None, > > > u'scheduled_at': None, u'node': None, u'ramdisk_id': u'', > > u'access_ip_v6': > > > None, u'access_ip_v4': None, u'kernel_id': u'', u'key_name': u'heat_key', > > > u'updated_at': None, u'host': None, u'user_id': > > > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'system_metadata': > > > {u'instance_type_memory_mb': 512, u'instance_type_swap': 0, > > > u'instance_type_vcpu_weight': None, u'instance_type_root_gb': 1, > > > u'instance_type_name': u'm1.tiny', u'instance_type_id': 2, > > > u'instance_type_ephemeral_gb': 0, u'instance_type_rxtx_factor': 1.0, > > > u'image_disk_format': u'qcow2', u'instance_type_flavorid': u'1', > > > u'instance_type_vcpus': 1, u'image_container_format': u'bare', > > > u'image_min_ram': 0, u'image_min_disk': 1, u'image_base_image_ref': > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'task_state': u'scheduling', > > > u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1, > > u'locked': > > > False, u'name': u'instance-00000112', u'created_at': > > > u'2014-02-24T08:41:08.257534', u'locked_by': None, u'launch_index': 0, > > > u'memory_mb': 512, u'vcpus': 1, u'image_ref': > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'architecture': None, > > > u'auto_disk_config': False, u'os_type': None, u'metadata': > > > {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': > > > u'tyky-Group_A-55cklit7nvbq', u'custom_metadata': u'test'}}, > > > u'security_group': [u'default'], u'instance_uuids': > > > [u'48c815ab-01c9-4ac8-9096-ac171976598c']}, u'event_type': > > > u'scheduler.run_instance.scheduled'}, u'counter_type': u'delta'} > > > > > > Also the following warnings/errors can be seen but they seem to occur > > when > > > auto scaling is properly working and have no negative effects as such: > > > > > > <44>Feb 24 08:43:08 node-16 > > > <U+FEFF>ceilometer-ceilometer.transformer.conversions WARNING: dropping > > > sample with no predecessor: <ceilometer.sample.Sample object at > > 0x3774fd0> > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <44>Feb 24 08:43:09 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: > > > Publishing 1 samples on metering > > > <43>Feb 24 08:43:09 node-16 > > > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to > > record > > > metering data: not okForStor > > > age > > > Traceback (most recent call last): > > > File > > > > > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", > > > line 65, in record_metering_data > > > self.storage_conn.record_metering_data(meter) > > > File > > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", > > > line 417, in record_metering_data > > > upsert=True, > > > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, > > in > > > update > > > check_keys, self.__uuid_subtype), safe) > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > > 969, in > > > _send_message > > > rv = self.__check_response_to_last_error(response) > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line > > 911, in > > > __check_response_to_last_error > > > raise OperationFailure(details["err"], details["code"]) > > > OperationFailure: not okForStorage > > > > > > Br, > > > -Juha > > > > > > _______________________________________________ > > > Mailing list: > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > Post to : openstack@lists.openstack.org > > > Unsubscribe : > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
