Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

[Ghanshyam Mann]

On Fri, Apr 21, 2017 at 4:19 AM, MONTEIRO, FELIPE C <fm5...@att.com> wrote:
> Thanks Ghanshyam,
>
> That was really helpful. If Tempest is versioned via tags, then I agree that 
> Patrole should do the same thing. I didn't quite know what you meant by 
> "feature flag" but after finding [0] it became apparent to me: Patrole has 
> been doing this wherever Tempest does it. Granted, we probably need more 
> participation in the project to catch minute details like that, as we 
> currently have a large body of tests, but currently we have been using 
> feature flags where appropriate.

Very Nice. Feel free to ping us if you need any help on those things.

>
> Case 3 as you mentioned is something that we've already had to account for: 
> When Nova and then Keystone moved their policies into code, we had to add 
> logic to the framework to account for those changes. Case 3 in theory would 
> not be a problem, if there was a way to verify whether a policy action was 
> enforced, other than by checking logs. If there was an API in OpenStack that 
> could be called to retrieve oslo_policy enforcements per request id, then 
> Case 3 would just result in an error being thrown by Patrole for a given 
> test, which we would then notice and fix.

Yea that's true and it was due to lack of policy documentation. Nova
will (almost completed) have good documentation to solve that. John &
OSIC folks started very nice documentation on policy which tells you
what all policy being used by what all APIs with description [1]. And
on same page Keystone also [2]

This is how it looks like now [3]. This can help Patrole.


..1 https://blueprints.launchpad.net/nova/+spec/policy-docs
..2 https://blueprints.launchpad.net/keystone/+spec/policy-docs
..3 https://docs.openstack.org/developer/nova/sample_policy.html
>
> [0] https://review.openstack.org/#/c/418928/
>
> --Felipe
>
> -Original Message-
> From: Ghanshyam Mann [mailto:ghanshyamm...@gmail.com]
> Sent: Wednesday, April 19, 2017 12:05 AM
> To: OpenStack Development Mailing List (not for usage questions) 
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole 
> release management
>
> On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C <fm5...@att.com> wrote:
>> Hi all,
>>
>>
>>
>> I have a question regarding patrole and release management. Many projects
>> like heat or murano have a tempest plugin within their repos, so by
>> extension their tempest plugins have releases, as the projects change over
>> time. However, since patrole is just a tempest plugin, yet heavily reliant
>> on tempest, how should patrole do release management?
>
> Hi Felipe,
>
> Release management depends on whether Patrole is planning the
> branchless model (i think it is) like Tempest or branch model., If
> branchless, then it does not fall under release management. It can
> adopt release model what Tempest has [1].
>
> Branchless model gives many benefit like avoid backward incompatible
> changes, avoid maintaining multiple Patrole repo across releases etc.
>
>> Am I correct in
>> thinking that it should, in the first place? With nova-network and other
>> APIs slated for deprecation in Pike and beyond, Patrole will logically have
>> to continuously be maintained to keep up, meaning that older tests, just
>> like with Tempest, will have to be phased out. If Patrole, then, does not
>> have releases, then older release-dependent tests and functionality will
>> over time be lost.
>
> We have 3 cases here:
> 1. Functionality is deprecated/removed in new release.
> 2. Functionality newly added in new release.
> 3. Policy enforcement change.
>
> For case 1, Tempest keep testing deprecated functionality till it is
> marked deprecated across all supported stable branches. Once all
> stable branch has that functionality as deprecated marked, then we
> discuss of removing its testing from Tempest.
> With API Microversion model that is little bit different where
> functionality might be deprecated after specific version or it is
> deprecated from base version itself. For example, Nova proxy APIs
> deprecated after 2.36, Certificate APIs might be gone from base
> version (which is under discussion). This is more case by case and
> based on all stack holder point of view, we will decide their testing
> should be removed or stay till when.
>
> For case 2, Tempest introduced testing of new functionality with
> feature flag and those tests will be executed/skipped accordingly.
>
> Case 3 is something important to consider for Patrole. Usually policy
> changes will be done with backward compatible way where changes does
> not break

Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

[MONTEIRO, FELIPE C]

Thanks Ghanshyam,

That was really helpful. If Tempest is versioned via tags, then I agree that 
Patrole should do the same thing. I didn't quite know what you meant by 
"feature flag" but after finding [0] it became apparent to me: Patrole has been 
doing this wherever Tempest does it. Granted, we probably need more 
participation in the project to catch minute details like that, as we currently 
have a large body of tests, but currently we have been using feature flags 
where appropriate.

Case 3 as you mentioned is something that we've already had to account for: 
When Nova and then Keystone moved their policies into code, we had to add logic 
to the framework to account for those changes. Case 3 in theory would not be a 
problem, if there was a way to verify whether a policy action was enforced, 
other than by checking logs. If there was an API in OpenStack that could be 
called to retrieve oslo_policy enforcements per request id, then Case 3 would 
just result in an error being thrown by Patrole for a given test, which we 
would then notice and fix.

[0] https://review.openstack.org/#/c/418928/ 

--Felipe

-Original Message-
From: Ghanshyam Mann [mailto:ghanshyamm...@gmail.com] 
Sent: Wednesday, April 19, 2017 12:05 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole release 
management

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C <fm5...@att.com> wrote:
> Hi all,
>
>
>
> I have a question regarding patrole and release management. Many projects
> like heat or murano have a tempest plugin within their repos, so by
> extension their tempest plugins have releases, as the projects change over
> time. However, since patrole is just a tempest plugin, yet heavily reliant
> on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

> Am I correct in
> thinking that it should, in the first place? With nova-network and other
> APIs slated for deprecation in Pike and beyond, Patrole will logically have
> to continuously be maintained to keep up, meaning that older tests, just
> like with Tempest, will have to be phased out. If Patrole, then, does not
> have releases, then older release-dependent tests and functionality will
> over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.


>
>
>
> Thank You,
>
> Felipe Monteiro

..1 
https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.openstack.org_wiki_QA_releases=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-SJ9DRnCxhze-aw=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY=9eq8-o16PTMYfNYW0LjlVyRfN1W4wXWloI7jVij5W4g=
 
..2 
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_391113_13=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-SJ9DRnCxhze-aw=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY=A5axQVIlZlVMGaW0vwQxpqwx5uMZvZ1ZNwWAXgrIoAc=
 

-gmann


>
>
>
>
> 

Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

[Ghanshyam Mann]

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C  wrote:
> Hi all,
>
>
>
> I have a question regarding patrole and release management. Many projects
> like heat or murano have a tempest plugin within their repos, so by
> extension their tempest plugins have releases, as the projects change over
> time. However, since patrole is just a tempest plugin, yet heavily reliant
> on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

> Am I correct in
> thinking that it should, in the first place? With nova-network and other
> APIs slated for deprecation in Pike and beyond, Patrole will logically have
> to continuously be maintained to keep up, meaning that older tests, just
> like with Tempest, will have to be phased out. If Patrole, then, does not
> have releases, then older release-dependent tests and functionality will
> over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.


>
>
>
> Thank You,
>
> Felipe Monteiro

..1 https://wiki.openstack.org/wiki/QA/releases
..2 https://review.openstack.org/#/c/391113/13

-gmann


>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa] [patrole] Question regarding patrole release management

[MONTEIRO, FELIPE C]

Hi all,

I have a question regarding patrole and release management. Many projects like 
heat or murano have a tempest plugin within their repos, so by extension their 
tempest plugins have releases, as the projects change over time. However, since 
patrole is just a tempest plugin, yet heavily reliant on tempest, how should 
patrole do release management? Am I correct in thinking that it should, in the 
first place? With nova-network and other APIs slated for deprecation in Pike 
and beyond, Patrole will logically have to continuously be maintained to keep 
up, meaning that older tests, just like with Tempest, will have to be phased 
out. If Patrole, then, does not have releases, then older release-dependent 
tests and functionality will over time be lost.

Thank You,
Felipe Monteiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev