Thanks Ghanshyam, That was really helpful. If Tempest is versioned via tags, then I agree that Patrole should do the same thing. I didn't quite know what you meant by "feature flag" but after finding [0] it became apparent to me: Patrole has been doing this wherever Tempest does it. Granted, we probably need more participation in the project to catch minute details like that, as we currently have a large body of tests, but currently we have been using feature flags where appropriate.
Case 3 as you mentioned is something that we've already had to account for: When Nova and then Keystone moved their policies into code, we had to add logic to the framework to account for those changes. Case 3 in theory would not be a problem, if there was a way to verify whether a policy action was enforced, other than by checking logs. If there was an API in OpenStack that could be called to retrieve oslo_policy enforcements per request id, then Case 3 would just result in an error being thrown by Patrole for a given test, which we would then notice and fix. [0] https://review.openstack.org/#/c/418928/ --Felipe -----Original Message----- From: Ghanshyam Mann [mailto:[email protected]] Sent: Wednesday, April 19, 2017 12:05 AM To: OpenStack Development Mailing List (not for usage questions) <[email protected]> Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C <[email protected]> wrote: > Hi all, > > > > I have a question regarding patrole and release management. Many projects > like heat or murano have a tempest plugin within their repos, so by > extension their tempest plugins have releases, as the projects change over > time. However, since patrole is just a tempest plugin, yet heavily reliant > on tempest, how should patrole do release management? Hi Felipe, Release management depends on whether Patrole is planning the branchless model (i think it is) like Tempest or branch model., If branchless, then it does not fall under release management. It can adopt release model what Tempest has [1]. Branchless model gives many benefit like avoid backward incompatible changes, avoid maintaining multiple Patrole repo across releases etc. > Am I correct in > thinking that it should, in the first place? With nova-network and other > APIs slated for deprecation in Pike and beyond, Patrole will logically have > to continuously be maintained to keep up, meaning that older tests, just > like with Tempest, will have to be phased out. If Patrole, then, does not > have releases, then older release-dependent tests and functionality will > over time be lost. We have 3 cases here: 1. Functionality is deprecated/removed in new release. 2. Functionality newly added in new release. 3. Policy enforcement change. For case 1, Tempest keep testing deprecated functionality till it is marked deprecated across all supported stable branches. Once all stable branch has that functionality as deprecated marked, then we discuss of removing its testing from Tempest. With API Microversion model that is little bit different where functionality might be deprecated after specific version or it is deprecated from base version itself. For example, Nova proxy APIs deprecated after 2.36, Certificate APIs might be gone from base version (which is under discussion). This is more case by case and based on all stack holder point of view, we will decide their testing should be removed or stay till when. For case 2, Tempest introduced testing of new functionality with feature flag and those tests will be executed/skipped accordingly. Case 3 is something important to consider for Patrole. Usually policy changes will be done with backward compatible way where changes does not break upgrade. Any change in policy enforcement will be done at least with supporting old and new rules [2] or with old rules deprecation of period of 1 release cycle at least. And branchless model can detect any accidental changes which going to break upgrade. IMO, branchless model is good value for Patrole in all 3 cases consideration and feature flags to handle the new/old/policy-change functionality. Similarly release model can be same as Tempest has. > > > > Thank You, > > Felipe Monteiro ..1 https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.openstack.org_wiki_QA_releases&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=9eq8-o16PTMYfNYW0LjlVyRfN1W4wXWloI7jVij5W4g&e= ..2 https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_391113_13&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=A5axQVIlZlVMGaW0vwQxpqwx5uMZvZ1ZNwWAXgrIoAc&e= -gmann > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e= > > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e= __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
