Re: [openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API
Help if you actually attach the link you want to send [0]. [0] https://bugs.launchpad.net/keystone/+bug/1702211 On 08/11/2017 11:26 AM, Morgan Fainberg wrote: > On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg > wrote: >> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro >> wrote: >>> Patrole tests occasionally fail while executing tests that test the >>> Identity v3 Extensions API [0]. Previously, this was not the case when >>> we used Fernet tokens and used a time.sleep(1) to allow for >>> role-switching to work correctly. However, we recently changed over to >>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a >>> time efficiency change. Ordinarily -- for well over 500 or so tests -- >>> this approach works successfully, with the exception of what appears >>> to be *random* v3 API extension tests [1][2] (random means different >>> tests pass or fail randomly across separate test runs). >>> >>> While there are a few solutions that come to mind on how to solve this >>> Patrole-side (like re-introducing a time.sleep() for specific APIs or >>> even avoiding role-switching altogether which is not as >>> straightforward as it sounds), we would still not understand *why* the >>> issue is happening in the first place. Is it a data-race condition? >>> Something specific to the identity v3 extensions API? A potential bug >>> or intended behavior somewhere? >>> >>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/ >>> [1] >>> http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group >>> [2] >>> http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906 >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> Are your tests causing token revocations? if so, there is a case where >> a revocation event is issued in the same second as a token (we've seen >> similar cases even in fernet) meaning the token is invalid when it is >> issued according to keystone. It's a long running bug. >> >> For the record, UUID tokens are deprecated and slated for removal in >> the R release. I recommend reverting to using Fernet tokens sooner >> rather than later. >> >> Last of all, the endpoint-filtering is generally not a great tool to >> use. I highly recommend not using it (or encouraging the use of it), >> it makes the catalog different depending on scope and provides zero >> added security benefit (anyone who knows the endpoint can use it >> anyway). > I am spinning up a change for Pike RC (right now) to address the > revocations occurring in the same second as the issuance of the token > (similar to a bug with password updates, which will also be fixed). > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API
More context on the patch Morgan is working on can be found in the bug report [0]. [0] On 08/11/2017 11:26 AM, Morgan Fainberg wrote: > On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg > wrote: >> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro >> wrote: >>> Patrole tests occasionally fail while executing tests that test the >>> Identity v3 Extensions API [0]. Previously, this was not the case when >>> we used Fernet tokens and used a time.sleep(1) to allow for >>> role-switching to work correctly. However, we recently changed over to >>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a >>> time efficiency change. Ordinarily -- for well over 500 or so tests -- >>> this approach works successfully, with the exception of what appears >>> to be *random* v3 API extension tests [1][2] (random means different >>> tests pass or fail randomly across separate test runs). >>> >>> While there are a few solutions that come to mind on how to solve this >>> Patrole-side (like re-introducing a time.sleep() for specific APIs or >>> even avoiding role-switching altogether which is not as >>> straightforward as it sounds), we would still not understand *why* the >>> issue is happening in the first place. Is it a data-race condition? >>> Something specific to the identity v3 extensions API? A potential bug >>> or intended behavior somewhere? >>> >>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/ >>> [1] >>> http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group >>> [2] >>> http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906 >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> Are your tests causing token revocations? if so, there is a case where >> a revocation event is issued in the same second as a token (we've seen >> similar cases even in fernet) meaning the token is invalid when it is >> issued according to keystone. It's a long running bug. >> >> For the record, UUID tokens are deprecated and slated for removal in >> the R release. I recommend reverting to using Fernet tokens sooner >> rather than later. >> >> Last of all, the endpoint-filtering is generally not a great tool to >> use. I highly recommend not using it (or encouraging the use of it), >> it makes the catalog different depending on scope and provides zero >> added security benefit (anyone who knows the endpoint can use it >> anyway). > I am spinning up a change for Pike RC (right now) to address the > revocations occurring in the same second as the issuance of the token > (similar to a bug with password updates, which will also be fixed). > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API
On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg wrote: > On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro > wrote: >> Patrole tests occasionally fail while executing tests that test the >> Identity v3 Extensions API [0]. Previously, this was not the case when >> we used Fernet tokens and used a time.sleep(1) to allow for >> role-switching to work correctly. However, we recently changed over to >> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a >> time efficiency change. Ordinarily -- for well over 500 or so tests -- >> this approach works successfully, with the exception of what appears >> to be *random* v3 API extension tests [1][2] (random means different >> tests pass or fail randomly across separate test runs). >> >> While there are a few solutions that come to mind on how to solve this >> Patrole-side (like re-introducing a time.sleep() for specific APIs or >> even avoiding role-switching altogether which is not as >> straightforward as it sounds), we would still not understand *why* the >> issue is happening in the first place. Is it a data-race condition? >> Something specific to the identity v3 extensions API? A potential bug >> or intended behavior somewhere? >> >> [0] https://developer.openstack.org/api-ref/identity/v3-ext/ >> [1] >> http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group >> [2] >> http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906 >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Are your tests causing token revocations? if so, there is a case where > a revocation event is issued in the same second as a token (we've seen > similar cases even in fernet) meaning the token is invalid when it is > issued according to keystone. It's a long running bug. > > For the record, UUID tokens are deprecated and slated for removal in > the R release. I recommend reverting to using Fernet tokens sooner > rather than later. > > Last of all, the endpoint-filtering is generally not a great tool to > use. I highly recommend not using it (or encouraging the use of it), > it makes the catalog different depending on scope and provides zero > added security benefit (anyone who knows the endpoint can use it > anyway). I am spinning up a change for Pike RC (right now) to address the revocations occurring in the same second as the issuance of the token (similar to a bug with password updates, which will also be fixed). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API
On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro wrote: > Patrole tests occasionally fail while executing tests that test the > Identity v3 Extensions API [0]. Previously, this was not the case when > we used Fernet tokens and used a time.sleep(1) to allow for > role-switching to work correctly. However, we recently changed over to > UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a > time efficiency change. Ordinarily -- for well over 500 or so tests -- > this approach works successfully, with the exception of what appears > to be *random* v3 API extension tests [1][2] (random means different > tests pass or fail randomly across separate test runs). > > While there are a few solutions that come to mind on how to solve this > Patrole-side (like re-introducing a time.sleep() for specific APIs or > even avoiding role-switching altogether which is not as > straightforward as it sounds), we would still not understand *why* the > issue is happening in the first place. Is it a data-race condition? > Something specific to the identity v3 extensions API? A potential bug > or intended behavior somewhere? > > [0] https://developer.openstack.org/api-ref/identity/v3-ext/ > [1] > http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group > [2] > http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906 > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Are your tests causing token revocations? if so, there is a case where a revocation event is issued in the same second as a token (we've seen similar cases even in fernet) meaning the token is invalid when it is issued according to keystone. It's a long running bug. For the record, UUID tokens are deprecated and slated for removal in the R release. I recommend reverting to using Fernet tokens sooner rather than later. Last of all, the endpoint-filtering is generally not a great tool to use. I highly recommend not using it (or encouraging the use of it), it makes the catalog different depending on scope and provides zero added security benefit (anyone who knows the endpoint can use it anyway). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
