Public bug reported: tempest.api.compute.servers.test_servers.ServerShowV263Test aims to test the optional trusted_image_certificates parameter introduced into n-api in microversion 2.63:
Add trusted_image_certificates to REST API https://review.opendev.org/#/c/486204/ Add new schema for Nova microversion 2.63 https://review.opendev.org/#/c/526485/ However this test currently relies on other tests having cached the image used to create the test instance before it is run. Otherwise the underlying logic within Nova [1] to validate the image using the provided certificates will be executed and fail as the image is missing image properties such as img_signature_hash_method etc [2]. [1] https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L402-L460 [2] https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_06_11_779 This behaviour can be easily reproduced in devstack by cleaning out the local image cache and running tempest.api.compute.servers.test_servers.ServerShowV263Test in isolation: $ rm -rf ~/data/nova/instances/_base/* $ tempest run --regex tempest.api.compute.servers.test_servers.ServerShowV263Test [..] ============================== Failed 1 tests - output below: ============================== tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server[id-71b8e3d5-11d2-494f-b917-b094a4afed3c] ------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "tempest/api/compute/servers/test_servers.py", line 194, in test_show_update_rebuild_list_server wait_until='ACTIVE') File "tempest/api/compute/base.py", line 246, in create_test_server **kwargs) File "tempest/common/compute.py", line 265, in create_test_server server['id']) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "tempest/common/compute.py", line 236, in create_test_server clients.servers_client, server['id'], wait_until) File "tempest/common/waiters.py", line 76, in wait_for_server_status server_id=server_id) tempest.exceptions.BuildErrorException: Server 254fb4a1-74a8-4681-ad67-a287bb3cde66 failed to build and is in ERROR status Details: {u'message': u'Build of instance 254fb4a1-74a8-4681-ad67-a287bb3cde66 aborted: Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-06-06T08:38:11Z'} Running all of the test_servers tests together however typically allows this to pass: $ rm -rf ~/data/nova/instances/_base/* $ tempest run --regex tempest.api.compute.servers.test_servers [..] ====== Totals ====== Ran: 59 tests in 150.0000 sec. - Passed: 58 - Skipped: 1 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 522.7800 sec. I'm going to tag nova into this bug report as well as I'm not sure that the current behaviour of n-api is correct. At present we attempt to verify the image when the optional trusted_image_certificates parameter is provided, regardless of CONF.glance.verify_glance_signatures being enabled within nova.conf: https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L414 While documented that behaviour doesn't seem right and I believe led to some incorrect assumptions being made by the initial writer of this test. https://docs.openstack.org/nova/latest/user/certificate-validation.html "Certificate validation will only be performed if image signature validation is enabled. However, the presence of trusted certificate IDs overrides the enable_certificate_validation and verify_glance_signatures settings. In other words, if a list of trusted certificate IDs is provided to the instance create or rebuild commands, signature verification and certificate validation will be performed, regardless of their settings in the Nova configurations. See Using Signature Verification for details." ** Affects: nova Importance: Undecided Status: New ** Affects: tempest Importance: Undecided Status: New ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1831866 Title: tempest.api.compute.servers.test_servers.ServerShowV263Test only passing when image has already been cached Status in OpenStack Compute (nova): New Status in tempest: New Bug description: tempest.api.compute.servers.test_servers.ServerShowV263Test aims to test the optional trusted_image_certificates parameter introduced into n-api in microversion 2.63: Add trusted_image_certificates to REST API https://review.opendev.org/#/c/486204/ Add new schema for Nova microversion 2.63 https://review.opendev.org/#/c/526485/ However this test currently relies on other tests having cached the image used to create the test instance before it is run. Otherwise the underlying logic within Nova [1] to validate the image using the provided certificates will be executed and fail as the image is missing image properties such as img_signature_hash_method etc [2]. [1] https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L402-L460 [2] https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_06_11_779 This behaviour can be easily reproduced in devstack by cleaning out the local image cache and running tempest.api.compute.servers.test_servers.ServerShowV263Test in isolation: $ rm -rf ~/data/nova/instances/_base/* $ tempest run --regex tempest.api.compute.servers.test_servers.ServerShowV263Test [..] ============================== Failed 1 tests - output below: ============================== tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server[id-71b8e3d5-11d2-494f-b917-b094a4afed3c] ------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "tempest/api/compute/servers/test_servers.py", line 194, in test_show_update_rebuild_list_server wait_until='ACTIVE') File "tempest/api/compute/base.py", line 246, in create_test_server **kwargs) File "tempest/common/compute.py", line 265, in create_test_server server['id']) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "tempest/common/compute.py", line 236, in create_test_server clients.servers_client, server['id'], wait_until) File "tempest/common/waiters.py", line 76, in wait_for_server_status server_id=server_id) tempest.exceptions.BuildErrorException: Server 254fb4a1-74a8-4681-ad67-a287bb3cde66 failed to build and is in ERROR status Details: {u'message': u'Build of instance 254fb4a1-74a8-4681-ad67-a287bb3cde66 aborted: Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-06-06T08:38:11Z'} Running all of the test_servers tests together however typically allows this to pass: $ rm -rf ~/data/nova/instances/_base/* $ tempest run --regex tempest.api.compute.servers.test_servers [..] ====== Totals ====== Ran: 59 tests in 150.0000 sec. - Passed: 58 - Skipped: 1 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 522.7800 sec. I'm going to tag nova into this bug report as well as I'm not sure that the current behaviour of n-api is correct. At present we attempt to verify the image when the optional trusted_image_certificates parameter is provided, regardless of CONF.glance.verify_glance_signatures being enabled within nova.conf: https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L414 While documented that behaviour doesn't seem right and I believe led to some incorrect assumptions being made by the initial writer of this test. https://docs.openstack.org/nova/latest/user/certificate- validation.html "Certificate validation will only be performed if image signature validation is enabled. However, the presence of trusted certificate IDs overrides the enable_certificate_validation and verify_glance_signatures settings. In other words, if a list of trusted certificate IDs is provided to the instance create or rebuild commands, signature verification and certificate validation will be performed, regardless of their settings in the Nova configurations. See Using Signature Verification for details." To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1831866/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
