Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Arkady_Kanevsky]

There is a version of Tempest that is released as part of OpenStack release.
Agree with Mark that we should stick to versions parity.

-Original Message-
From: Mark Voelker [mailto:mvoel...@vmware.com]
Sent: Monday, June 20, 2016 8:25 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable 
some strict response checking for interop testing


> On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote:
>
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
>
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>>
>>
>>> I don't think DefCore actually needs to change old versions of
>>> Tempest, but maybe Chris or Mark can verify that?
>>
>> So if I'm groking this correctly, there's kind of two scenarios being
>> painted here. One is the "LCD" approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline. The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline. The
>> former may result in some fairly long-lived flags, and the latter is
>> actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>>
>> In some cases, tests get flagged in the Guidelines because of bugs in
>> the test or because the test needs refactoring. The underlying
>> Capabilities the those tests are testing actually work fine. Once we
>> identify such an issue, the test can be fixed...in master. Under the
>> first scenario, this potentially creates some very long-lived flags:
>>
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released). It's one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud. They run the Juno EOL version of Tempest (tag=8), they
>> find a test issue, and we flag it. A few weeks later, a fix lands in
>> Tempest. Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest. That doesn't have the
>> fix, so the flag stays. Another six months goes by and we get a
>> Guideline and we're up to the Liberty-EOL version of Tempest. No
>> fix, flag stays. Six more months, and now we're at Mitaka-EOL, and
>> that's the first version that includes the fix.
>>
>> Generally speaking long lived flags aren't so great because it means
>> the tests are not required...which means there's less or no assurance
>> that the capabilities they test for actually work in the clouds that
>> adhere to those Guidelines. So, the worst-case scenario here looks
>> kind of ugly.
>>
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API's that are long-lived
>> across many releases, so we haven't run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date. In fact
>> I'm struggling to think of a time we've flagged something because
>> someone complained the test wasn't runnable against an older release
>> covered by the Guideline in question. I can think of plenty of times
>> where we've flagged something due to a test issue though...keep in mind
>> we're still in pretty formative times with DefCore here where these
>> tests are starting to be used in a new way for the first time.
>> Anyway, as Matt points out we could potentially use a much newer
>> Tempest tag: tag=11 (which is the start of Newton development and is
>> a roughly 2 month old version of Tempest). Next Guideline rolls
>> around, we use the tag for start-of-ocata, and we get the fix and can
>> drop the flag.
>>
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past, and
>> still can). When we see a fix for a flagged test go in, we or the
>> Refstack folks can do a quick test to make sure everything's in order
>> and then update that SHA to match the version with the fix. That way
>> we're relatively sure we have a version that works today, and will
>> work when we drop the flag in the next Guideline too. When we
>> finalize that next Guideline, we also update the test-repositories
>> section of the new Guideline that Matt pointed to earlier to reflect
>> the best-known version on the da

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Arkady_Kanevsky]

Chris,
If we add openstack config to refstack submission will that provide sufficient 
info for "interoperability" LOGO?
That includes version  of APIs for each service.
https://review.openstack.org/#/c/300057/

Thanks,
Arkady

-Original Message-
From: Chris Hoge [mailto:ch...@openstack.org]
Sent: Wednesday, June 22, 2016 1:37 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable 
some strict response checking for interop testing


> On Jun 22, 2016, at 11:24 AM, Sean Dague wrote:
>
> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>>
>>> On Jun 20, 2016, at 5:10 AM, Sean Dague
>>> > wrote:
>>>
>>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>>>
>>>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe
>>>>> > wrote:
>>>>>
>>>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish
>>>>> > wrote:
>>>>>
>>>>>> But, if we add another possible state on the defcore side like
>>>>>> conditional pass, warning, yellow, etc. (the name doesn't matter)
>>>>>> which is used to indicate that things on product X could only
>>>>>> pass when strict validation was disabled (and be clear about
>>>>>> where and why) then my concerns would be alleviated.
>>>>>> I just do
>>>>>> not want this to end up not being visible to end users trying to
>>>>>> evaluate interoperability of different clouds using the test
>>>>>> results.
>>>>>
>>>>> +1
>>>>>
>>>>> Don't fail them, but don't cover up their incompatibility, either.
>>>>> -- Ed Leafe
>>>>
>>>> That's not my proposal. My requirement is that vendors who want to
>>>> do this state exactly which APIs are sending back additional data,
>>>> and that this information be published.
>>>>
>>>> There are different levels of incompatibility. A response with
>>>> additional data that can be safely ignored is different from a
>>>> changed response that would cause a client to fail.
>>>
>>> It's actually not different. It's really not.
>>>
>>> This idea that it's safe to add response data is based on an
>>> assumption that software versions only move forward. If you have a
>>> single deploy of software, that's fine.
>>>
>>> However as noted, we've got production clouds on Juno <-> Mitaka in
>>> the wild. Which means if we want to support horizontal transfer
>>> between clouds, the user experienced timeline might be start on a
>>> Mitaka cloud, then try to move to Juno. So anything added from Juno
>>> -> Mitaka without signaling has exactly the same client breaking
>>> behavior as removing attributes.
>>>
>>> Which is why microversions are needed for attribute adds.
>>
>> I'd like to note that Nova v2.0 is still a supported API, which as
>> far as I understand allows for additional attributes and extensions.
>> That Tempest doesn't allow for disabling strict checking when using a
>> v2.0 endpoint is a problem.
>>
>> The reporting of v2.0 in the Marketplace (which is what we do right
>> now) is also a signal to a user that there may be vendor additions to
>> the API.
>>
>> DefCore doesn't disallow the use of a 2.0 endpoint as part of the
>> interoperability standard.
>
> This is a point of confusion.
>
> The API definition did not allow that. The implementation of the API
> stack did.

And downstream vendors took advantage of that. We may not like it, but it's a 
reality in the current ecosystem.

> In Liberty the v2.0 API is optionally provided by a different backend
> stack that doesn't support extensions.
> In Mitaka it is default v2.0 API on a non extensions backend In Newton
> the old backend is deleted.
>
> From Newton forward there is still a v2.0 API, but all the code hooks
> that provided facilities for extensions are gone.

It's really important that the current documentation reflect the code and 
intent of the dev team. As of writing this e-mail,

"* v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will be deprecated in the near 
future.)"[1]

Even with this being removed in Newton, DefCore still has to allow for it in 
every supported version.

-Chris

[1] http://docs.openstack.org/developer/nova/

> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 03:13 PM, Jay Faulkner wrote:
> 
> 
> On 6/22/16 12:01 PM, Sean Dague wrote:
>> On 06/22/2016 02:37 PM, Chris Hoge wrote:
 On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:

 On 06/22/2016 01:59 PM, Chris Hoge wrote:
>> On Jun 20, 2016, at 5:10 AM, Sean Dague > > wrote:
>>
>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
 On Jun 14, 2016, at 3:59 PM, Edward Leafe > wrote:

 On Jun 14, 2016, at 5:50 PM, Matthew Treinish > wrote:

> But, if we add another possible state on the defcore side like
> conditional pass,
> warning, yellow, etc. (the name doesn't matter) which is used to
> indicate that
> things on product X could only pass when strict validation was
> disabled (and
> be clear about where and why) then my concerns would be alleviated.
> I just do
> not want this to end up not being visible to end users trying to
> evaluate
> interoperability of different clouds using the test results.
 +1

 Don't fail them, but don't cover up their incompatibility, either.
 -- Ed Leafe
>>> That’s not my proposal. My requirement is that vendors who want to do
>>> this
>>> state exactly which APIs are sending back additional data, and that this
>>> information be published.
>>>
>>> There are different levels of incompatibility. A response with
>>> additional data
>>> that can be safely ignored is different from a changed response that
>>> would
>>> cause a client to fail.
>> It's actually not different. It's really not.
>>
>> This idea that it's safe to add response data is based on an assumption
>> that software versions only move forward. If you have a single deploy of
>> software, that's fine.
>>
>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>> wild. Which means if we want to support horizontal transfer between
>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>> then try to move to Juno. So anything added from Juno -> Mitaka without
>> signaling has exactly the same client breaking behavior as removing
>> attributes.
>>
>> Which is why microversions are needed for attribute adds.
> I’d like to note that Nova v2.0 is still a supported API, which
> as far as I understand allows for additional attributes and
> extensions. That Tempest doesn’t allow for disabling strict
> checking when using a v2.0 endpoint is a problem.
>
> The reporting of v2.0 in the Marketplace (which is what we do
> right now) is also a signal to a user that there may be vendor
> additions to the API.
>
> DefCore doesn’t disallow the use of a 2.0 endpoint as part
> of the interoperability standard.
 This is a point of confusion.

 The API definition did not allow that. The implementation of the API
 stack did.
>>> And downstream vendors took advantage of that. We may
>>> not like it, but it’s a reality in the current ecosystem.
>> And we started saying "stop it" 2 years ago. And we've consistently been
>> saying stop it all along. And now it's gone.
>>
>> And yes, for people that did not get ahead of this issue and engage the
>> community, it now hurts. But this has been a quite long process.
> I don't wanna wade fully into this discussion, but a question about this
> here as there seems to be somewhat of a double standard. I know
> upstream, we generally "pay the price" for bad API decisions almost
> indefinitely, because we don't want to break users. 

Upstream pays the upstream API cost. Because that's what's in upstream
and because that's what the API is defined as.

> Is it reasonable to
> expecting a public/vendor cloud, who will typically has even more
> change-averse users, to change that API out from under users without a
> version bump?

What you are asking is that upstream pays the fork cost of every public
cloud forever because they didn't collaborate upstream for their needs.

That can't be the right answer in Open Source, ever. It immediately
makes upstream actually impossible to fix root issues. Like security
issues, which is a huge reason that we put the strict inbound validation
into the API stack, as it means the data we're acting on has a
sanitization layer.

If your feature is not in tree, and part of the upstream testing, it is
at risk of break at any time. Upstream is not just a salt mine you
extract free stuff from, it's a garden you need to participate in and
sow and harvest together.

Which, again, is why we've been having this conversation for *2 years*
in the open. There was a 2 year coasting window to get there, to propose
the required elements into upstream, and 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Jay Faulkner]

On 6/22/16 12:01 PM, Sean Dague wrote:
> On 06/22/2016 02:37 PM, Chris Hoge wrote:
>>> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
>>>
>>> On 06/22/2016 01:59 PM, Chris Hoge wrote:
> On Jun 20, 2016, at 5:10 AM, Sean Dague  > wrote:
>
> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe >> > wrote:
>>>
>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish >> > wrote:
>>>
 But, if we add another possible state on the defcore side like
 conditional pass,
 warning, yellow, etc. (the name doesn't matter) which is used to
 indicate that
 things on product X could only pass when strict validation was
 disabled (and
 be clear about where and why) then my concerns would be alleviated.
 I just do
 not want this to end up not being visible to end users trying to
 evaluate
 interoperability of different clouds using the test results.
>>> +1
>>>
>>> Don't fail them, but don't cover up their incompatibility, either.
>>> -- Ed Leafe
>> That’s not my proposal. My requirement is that vendors who want to do
>> this
>> state exactly which APIs are sending back additional data, and that this
>> information be published.
>>
>> There are different levels of incompatibility. A response with
>> additional data
>> that can be safely ignored is different from a changed response that
>> would
>> cause a client to fail.
> It's actually not different. It's really not.
>
> This idea that it's safe to add response data is based on an assumption
> that software versions only move forward. If you have a single deploy of
> software, that's fine.
>
> However as noted, we've got production clouds on Juno <-> Mitaka in the
> wild. Which means if we want to support horizontal transfer between
> clouds, the user experienced timeline might be start on a Mitaka cloud,
> then try to move to Juno. So anything added from Juno -> Mitaka without
> signaling has exactly the same client breaking behavior as removing
> attributes.
>
> Which is why microversions are needed for attribute adds.
 I’d like to note that Nova v2.0 is still a supported API, which
 as far as I understand allows for additional attributes and
 extensions. That Tempest doesn’t allow for disabling strict
 checking when using a v2.0 endpoint is a problem.

 The reporting of v2.0 in the Marketplace (which is what we do
 right now) is also a signal to a user that there may be vendor
 additions to the API.

 DefCore doesn’t disallow the use of a 2.0 endpoint as part
 of the interoperability standard.
>>> This is a point of confusion.
>>>
>>> The API definition did not allow that. The implementation of the API
>>> stack did.
>> And downstream vendors took advantage of that. We may
>> not like it, but it’s a reality in the current ecosystem.
> And we started saying "stop it" 2 years ago. And we've consistently been
> saying stop it all along. And now it's gone.
>
> And yes, for people that did not get ahead of this issue and engage the
> community, it now hurts. But this has been a quite long process.
I don't wanna wade fully into this discussion, but a question about this
here as there seems to be somewhat of a double standard. I know
upstream, we generally "pay the price" for bad API decisions almost
indefinitely, because we don't want to break users. Is it reasonable to
expecting a public/vendor cloud, who will typically has even more
change-averse users, to change that API out from under users without a
version bump?

-Jay
>>> In Liberty the v2.0 API is optionally provided by a different backend
>>> stack that doesn't support extensions.
>>> In Mitaka it is default v2.0 API on a non extensions backend
>>> In Newton the old backend is deleted.
>>>
>>> From Newton forward there is still a v2.0 API, but all the code hooks
>>> that provided facilities for extensions are gone.
>> It’s really important that the current documentation reflect the
>> code and intent of the dev team. As of writing this e-mail, 
>>
>> "• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
>> be deprecated in the near future.)”[1]
>>
>> Even with this being removed in Newton, DefCore still has
>> to allow for it in every supported version.
> The v2 extensions link there, you will notice, is upstream extensions.
> All of which default on for the new code stack.
>
> Everything documented there still works on the new code stack. The v2 +
> v2 extensions linked there remains supported in Newton.
>
> The wording on this page should be updated, it is in the Nova developer
> docs, intended for people working on Nova upstream. They lag a bit from
> where 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 02:37 PM, Chris Hoge wrote:
> 
>> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
>>
>> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>>>
 On Jun 20, 2016, at 5:10 AM, Sean Dague > wrote:

 On 06/14/2016 07:19 PM, Chris Hoge wrote:
>
>> On Jun 14, 2016, at 3:59 PM, Edward Leafe > > wrote:
>>
>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish > > wrote:
>>
>>> But, if we add another possible state on the defcore side like
>>> conditional pass,
>>> warning, yellow, etc. (the name doesn't matter) which is used to
>>> indicate that
>>> things on product X could only pass when strict validation was
>>> disabled (and
>>> be clear about where and why) then my concerns would be alleviated.
>>> I just do
>>> not want this to end up not being visible to end users trying to
>>> evaluate
>>> interoperability of different clouds using the test results.
>>
>> +1
>>
>> Don't fail them, but don't cover up their incompatibility, either.
>> -- Ed Leafe
>
> That’s not my proposal. My requirement is that vendors who want to do
> this
> state exactly which APIs are sending back additional data, and that this
> information be published.
>
> There are different levels of incompatibility. A response with
> additional data
> that can be safely ignored is different from a changed response that
> would
> cause a client to fail.

 It's actually not different. It's really not.

 This idea that it's safe to add response data is based on an assumption
 that software versions only move forward. If you have a single deploy of
 software, that's fine.

 However as noted, we've got production clouds on Juno <-> Mitaka in the
 wild. Which means if we want to support horizontal transfer between
 clouds, the user experienced timeline might be start on a Mitaka cloud,
 then try to move to Juno. So anything added from Juno -> Mitaka without
 signaling has exactly the same client breaking behavior as removing
 attributes.

 Which is why microversions are needed for attribute adds.
>>>
>>> I’d like to note that Nova v2.0 is still a supported API, which
>>> as far as I understand allows for additional attributes and
>>> extensions. That Tempest doesn’t allow for disabling strict
>>> checking when using a v2.0 endpoint is a problem.
>>>
>>> The reporting of v2.0 in the Marketplace (which is what we do
>>> right now) is also a signal to a user that there may be vendor
>>> additions to the API.
>>>
>>> DefCore doesn’t disallow the use of a 2.0 endpoint as part
>>> of the interoperability standard.
>>
>> This is a point of confusion.
>>
>> The API definition did not allow that. The implementation of the API
>> stack did.
> 
> And downstream vendors took advantage of that. We may
> not like it, but it’s a reality in the current ecosystem.

And we started saying "stop it" 2 years ago. And we've consistently been
saying stop it all along. And now it's gone.

And yes, for people that did not get ahead of this issue and engage the
community, it now hurts. But this has been a quite long process.

>> In Liberty the v2.0 API is optionally provided by a different backend
>> stack that doesn't support extensions.
>> In Mitaka it is default v2.0 API on a non extensions backend
>> In Newton the old backend is deleted.
>>
>> From Newton forward there is still a v2.0 API, but all the code hooks
>> that provided facilities for extensions are gone.
> 
> It’s really important that the current documentation reflect the
> code and intent of the dev team. As of writing this e-mail, 
> 
> "• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
> be deprecated in the near future.)”[1]
> 
> Even with this being removed in Newton, DefCore still has
> to allow for it in every supported version.

The v2 extensions link there, you will notice, is upstream extensions.
All of which default on for the new code stack.

Everything documented there still works on the new code stack. The v2 +
v2 extensions linked there remains supported in Newton.

The wording on this page should be updated, it is in the Nova developer
docs, intended for people working on Nova upstream. They lag a bit from
where reality is, as does documentation everywhere.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
> 
> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>> 
>>> On Jun 20, 2016, at 5:10 AM, Sean Dague >> > wrote:
>>> 
>>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
 
> On Jun 14, 2016, at 3:59 PM, Edward Leafe  > wrote:
> 
> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  > wrote:
> 
>> But, if we add another possible state on the defcore side like
>> conditional pass,
>> warning, yellow, etc. (the name doesn't matter) which is used to
>> indicate that
>> things on product X could only pass when strict validation was
>> disabled (and
>> be clear about where and why) then my concerns would be alleviated.
>> I just do
>> not want this to end up not being visible to end users trying to
>> evaluate
>> interoperability of different clouds using the test results.
> 
> +1
> 
> Don't fail them, but don't cover up their incompatibility, either.
> -- Ed Leafe
 
 That’s not my proposal. My requirement is that vendors who want to do
 this
 state exactly which APIs are sending back additional data, and that this
 information be published.
 
 There are different levels of incompatibility. A response with
 additional data
 that can be safely ignored is different from a changed response that
 would
 cause a client to fail.
>>> 
>>> It's actually not different. It's really not.
>>> 
>>> This idea that it's safe to add response data is based on an assumption
>>> that software versions only move forward. If you have a single deploy of
>>> software, that's fine.
>>> 
>>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>>> wild. Which means if we want to support horizontal transfer between
>>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>>> then try to move to Juno. So anything added from Juno -> Mitaka without
>>> signaling has exactly the same client breaking behavior as removing
>>> attributes.
>>> 
>>> Which is why microversions are needed for attribute adds.
>> 
>> I’d like to note that Nova v2.0 is still a supported API, which
>> as far as I understand allows for additional attributes and
>> extensions. That Tempest doesn’t allow for disabling strict
>> checking when using a v2.0 endpoint is a problem.
>> 
>> The reporting of v2.0 in the Marketplace (which is what we do
>> right now) is also a signal to a user that there may be vendor
>> additions to the API.
>> 
>> DefCore doesn’t disallow the use of a 2.0 endpoint as part
>> of the interoperability standard.
> 
> This is a point of confusion.
> 
> The API definition did not allow that. The implementation of the API
> stack did.

And downstream vendors took advantage of that. We may
not like it, but it’s a reality in the current ecosystem.

> In Liberty the v2.0 API is optionally provided by a different backend
> stack that doesn't support extensions.
> In Mitaka it is default v2.0 API on a non extensions backend
> In Newton the old backend is deleted.
> 
> From Newton forward there is still a v2.0 API, but all the code hooks
> that provided facilities for extensions are gone.

It’s really important that the current documentation reflect the
code and intent of the dev team. As of writing this e-mail, 

"• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
be deprecated in the near future.)”[1]

Even with this being removed in Newton, DefCore still has
to allow for it in every supported version.

-Chris

[1] http://docs.openstack.org/developer/nova/

>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 01:59 PM, Chris Hoge wrote:
> 
>> On Jun 20, 2016, at 5:10 AM, Sean Dague > > wrote:
>>
>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>>
 On Jun 14, 2016, at 3:59 PM, Edward Leafe > wrote:

 On Jun 14, 2016, at 5:50 PM, Matthew Treinish > wrote:

> But, if we add another possible state on the defcore side like
> conditional pass,
> warning, yellow, etc. (the name doesn't matter) which is used to
> indicate that
> things on product X could only pass when strict validation was
> disabled (and
> be clear about where and why) then my concerns would be alleviated.
> I just do
> not want this to end up not being visible to end users trying to
> evaluate
> interoperability of different clouds using the test results.

 +1

 Don't fail them, but don't cover up their incompatibility, either.
 -- Ed Leafe
>>>
>>> That’s not my proposal. My requirement is that vendors who want to do
>>> this
>>> state exactly which APIs are sending back additional data, and that this
>>> information be published.
>>>
>>> There are different levels of incompatibility. A response with
>>> additional data
>>> that can be safely ignored is different from a changed response that
>>> would
>>> cause a client to fail.
>>
>> It's actually not different. It's really not.
>>
>> This idea that it's safe to add response data is based on an assumption
>> that software versions only move forward. If you have a single deploy of
>> software, that's fine.
>>
>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>> wild. Which means if we want to support horizontal transfer between
>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>> then try to move to Juno. So anything added from Juno -> Mitaka without
>> signaling has exactly the same client breaking behavior as removing
>> attributes.
>>
>> Which is why microversions are needed for attribute adds.
> 
> I’d like to note that Nova v2.0 is still a supported API, which
> as far as I understand allows for additional attributes and
> extensions. That Tempest doesn’t allow for disabling strict
> checking when using a v2.0 endpoint is a problem.
> 
> The reporting of v2.0 in the Marketplace (which is what we do
> right now) is also a signal to a user that there may be vendor
> additions to the API.
> 
> DefCore doesn’t disallow the use of a 2.0 endpoint as part
> of the interoperability standard.

This is a point of confusion.

The API definition did not allow that. The implementation of the API
stack did.

In Liberty the v2.0 API is optionally provided by a different backend
stack that doesn't support extensions.
In Mitaka it is default v2.0 API on a non extensions backend
In Newton the old backend is deleted.

From Newton forward there is still a v2.0 API, but all the code hooks
that provided facilities for extensions are gone.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 20, 2016, at 5:10 AM, Sean Dague  wrote:
> 
> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>> 
>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
>>> 
>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
>>> 
 But, if we add another possible state on the defcore side like conditional 
 pass,
 warning, yellow, etc. (the name doesn't matter) which is used to indicate 
 that
 things on product X could only pass when strict validation was disabled 
 (and
 be clear about where and why) then my concerns would be alleviated. I just 
 do
 not want this to end up not being visible to end users trying to evaluate
 interoperability of different clouds using the test results.
>>> 
>>> +1
>>> 
>>> Don't fail them, but don't cover up their incompatibility, either.
>>> -- Ed Leafe
>> 
>> That’s not my proposal. My requirement is that vendors who want to do this
>> state exactly which APIs are sending back additional data, and that this
>> information be published.
>> 
>> There are different levels of incompatibility. A response with additional 
>> data
>> that can be safely ignored is different from a changed response that would
>> cause a client to fail.
> 
> It's actually not different. It's really not.
> 
> This idea that it's safe to add response data is based on an assumption
> that software versions only move forward. If you have a single deploy of
> software, that's fine.
> 
> However as noted, we've got production clouds on Juno <-> Mitaka in the
> wild. Which means if we want to support horizontal transfer between
> clouds, the user experienced timeline might be start on a Mitaka cloud,
> then try to move to Juno. So anything added from Juno -> Mitaka without
> signaling has exactly the same client breaking behavior as removing
> attributes.
> 
> Which is why microversions are needed for attribute adds.

I’d like to note that Nova v2.0 is still a supported API, which
as far as I understand allows for additional attributes and
extensions. That Tempest doesn’t allow for disabling strict
checking when using a v2.0 endpoint is a problem.

The reporting of v2.0 in the Marketplace (which is what we do
right now) is also a signal to a user that there may be vendor
additions to the API.

DefCore doesn’t disallow the use of a 2.0 endpoint as part
of the interoperability standard.

-Chris


>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 20, 2016, at 6:56 AM, Doug Hellmann  wrote:
> 

> 
> I'm also, I think, edging away from the "we need to find a compromise"
> camp into the "why is this turning into such a big deal" camp. How did
> we get into a situation where the community has set a clear direction,
> the trademark certification system has a long lead time built in, and
> vendors are still not able to maintain certification?

If I had to make an educated guess, it’s because product managers
have to produce a roadmap with goals and features that consider
both what’s happening upstream, what is currently deployed, and
existing customers.

Just pulling attributes that were once ‘ok’ within the ecosystem and
now aren’t (even with lead time) isn’t as easy as “just change the
response”. It takes time, and given the year-long cycle that DefCore
has adopted for re-testing and the relative youth of the OpenStack
Powered Trademark program, it’s not unsurprising that a few
clouds have been hit by this.

I’ve spoken with all three vendors who are being hit by this, and
two definitely have a longer lead time to work on this. The third
is using the extra responses internally and are currently working
on changing their custom tools to get the extra information in
a different way.

I’ve also spoken with another vendor who is going to be caught
by it, though, and have explained the situation to them and
they are considering their options.

I am now actively telling vendors who are still sending additional
properties to start with with the Nova team upstream to have
their additional properties micro-versioned.

> Are the systems
> running modified versions of newer OpenStack that can't be certified
> with older versions of Tempest?

With the volume of bug fixes, refactoring, and general improvements
from the last year, I’d say no.

-Chris

> 
> Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Mark Voelker's message of 2016-06-20 13:24:46 +:
> 
> > On Jun 20, 2016, at 8:46 AM, Doug Hellmann  wrote:
> > 
> > Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
> > 
> >> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> >> 
> >> 
> >>> I don't think DefCore actually needs to change old versions of Tempest,
> >>> but maybe Chris or Mark can verify that?
> >> 
> >> So if I’m groking this correctly, there’s kind of two scenarios
> >> being painted here.  One is the “LCD” approach where we use the
> >> $osversion-eol version of Tempest, where $osversion matches the
> >> oldest version covered in a Guideline.  The other is to use the
> >> start-of-$osversion version of Tempest where $osversion is the
> >> OpenStack version after the most recent one in the Guideline.  The
> >> former may result in some fairly long-lived flags, and the latter
> >> is actually not terribly different than what we do today I think.
> >> Let me try to talk through both...
> >> 
> >> In some cases, tests get flagged in the Guidelines because of
> >> bugs in the test or because the test needs refactoring.  The
> >> underlying Capabilities the those tests are testing actually work
> >> fine.  Once we identify such an issue, the test can be fixed…in
> >> master.  Under the first scenario, this potentially creates some
> >> very long-lived flags:
> >> 
> >> 2016.01 is the most current Guideline right now covers Juno, Kilo,
> >> Liberty (and Mitaka after it was released).  It’s one of the two
> >> Guidelines that you can use if you want an OpenStack Powered license
> >> from he Foundation, $vendor wants to run it against their shiny new
> >> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
> >> they find a test issue, and we flag it.  A few weeks later, a fix
> >> lands in Tempest.  Several months later the next Guideline rolls
> >> around: the oldest covered release is Kilo and we start telling
> >> people to use the Kilo-EOL version of Tempest.  That doesn’t have
> >> the fix, so the flag stays.  Another six months goes by and we get
> >> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
> >> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
> >> that's the first version that includes the fix.
> >> 
> >> Generally speaking long lived flags aren’t so great because it
> >> means the tests are not required…which means there’s less or no
> >> assurance that the capabilities they test for actually work in the
> >> clouds that adhere to those Guidelines.  So, the worst-case scenario
> >> here looks kind of ugly.
> >> 
> >> As Matt correctly pointed out though, the capabilities DefCore
> >> selects for are generally pretty stable API’s that are long-lived
> >> across many releases, so we haven’t run into a lot of issues running
> >> pretty new versions of Tempest against older clouds to date.  In
> >> fact I’m struggling to think of a time we’ve flagged something
> >> because someone complained the test wasn’t runnable against an older
> >> release covered by the Guideline in question.  I can think of plenty
> >> of times where we’ve flagged something due to a test issue though…keep
> >> in mind we’re still in pretty formative times with DefCore here
> >> where these tests are starting to be used in a new way for the first
> >> time.  Anyway, as Matt points out we could potentially use a much
> >> newer Tempest tag: tag=11 (which is the start of Newton development
> >> and is a roughly 2 month old version of Tempest).  Next Guideline
> >> rolls around, we use the tag for start-of-ocata, and we get the fix
> >> and can drop the flag.
> >> 
> >> Today, RefStack client by default checks out a specific SHA of
> >> Tempest [1] (it actually did use a tag at some point in the past,
> >> and still can).  When we see a fix for a flagged test go in, we or
> >> the Refstack folks can do a quick test to make sure everything’s
> >> in order and then update that SHA to match the version with the
> >> fix.  That way we’re relatively sure we have a version that works
> >> today, and will work when we drop the flag in the next Guideline
> >> too.  When we finalize that next Guideline, we also update the
> >> test-repositories section of the new Guideline that Matt pointed
> >> to earlier to reflect the best-known version on the day the Guideline
> >> was sent to the Board for approval.  One added benefit of this
> >> approach is that people running the tests today may get a version
> >> of Tempest that includes a fix for a flagged test.  A flagged test
> >> isn’t required, but it does get run—and now will show a passing
> >> result, so we have data that says “this provider actually does
> >> support this capability (even though it’s flagged), and the test
> >> does indeed seem to be working."
> >> 
> >> 
> >> So, that’s actually not hugely different from the second scenario
> >> I think?  Or did I miss something there?
> > 
> > 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

> On Jun 20, 2016, at 8:46 AM, Doug Hellmann  wrote:
> 
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
> 
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> 
>> 
>>> I don't think DefCore actually needs to change old versions of Tempest,
>>> but maybe Chris or Mark can verify that?
>> 
>> So if I’m groking this correctly, there’s kind of two scenarios
>> being painted here.  One is the “LCD” approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline.  The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline.  The
>> former may result in some fairly long-lived flags, and the latter
>> is actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>> 
>> In some cases, tests get flagged in the Guidelines because of
>> bugs in the test or because the test needs refactoring.  The
>> underlying Capabilities the those tests are testing actually work
>> fine.  Once we identify such an issue, the test can be fixed…in
>> master.  Under the first scenario, this potentially creates some
>> very long-lived flags:
>> 
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released).  It’s one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
>> they find a test issue, and we flag it.  A few weeks later, a fix
>> lands in Tempest.  Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest.  That doesn’t have
>> the fix, so the flag stays.  Another six months goes by and we get
>> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
>> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
>> that's the first version that includes the fix.
>> 
>> Generally speaking long lived flags aren’t so great because it
>> means the tests are not required…which means there’s less or no
>> assurance that the capabilities they test for actually work in the
>> clouds that adhere to those Guidelines.  So, the worst-case scenario
>> here looks kind of ugly.
>> 
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API’s that are long-lived
>> across many releases, so we haven’t run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date.  In
>> fact I’m struggling to think of a time we’ve flagged something
>> because someone complained the test wasn’t runnable against an older
>> release covered by the Guideline in question.  I can think of plenty
>> of times where we’ve flagged something due to a test issue though…keep
>> in mind we’re still in pretty formative times with DefCore here
>> where these tests are starting to be used in a new way for the first
>> time.  Anyway, as Matt points out we could potentially use a much
>> newer Tempest tag: tag=11 (which is the start of Newton development
>> and is a roughly 2 month old version of Tempest).  Next Guideline
>> rolls around, we use the tag for start-of-ocata, and we get the fix
>> and can drop the flag.
>> 
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past,
>> and still can).  When we see a fix for a flagged test go in, we or
>> the Refstack folks can do a quick test to make sure everything’s
>> in order and then update that SHA to match the version with the
>> fix.  That way we’re relatively sure we have a version that works
>> today, and will work when we drop the flag in the next Guideline
>> too.  When we finalize that next Guideline, we also update the
>> test-repositories section of the new Guideline that Matt pointed
>> to earlier to reflect the best-known version on the day the Guideline
>> was sent to the Board for approval.  One added benefit of this
>> approach is that people running the tests today may get a version
>> of Tempest that includes a fix for a flagged test.  A flagged test
>> isn’t required, but it does get run—and now will show a passing
>> result, so we have data that says “this provider actually does
>> support this capability (even though it’s flagged), and the test
>> does indeed seem to be working."
>> 
>> 
>> So, that’s actually not hugely different from the second scenario
>> I think?  Or did I miss something there?
> 
> What I was proposing is that we keep the certification rules in
> sync with OpenStack versions. If a vendor stays on an old version
> of OpenStack, they can keep using the old (matching) version of
> Tempest to certify.  It's not clear from 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:

> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> 
> 
> > I don't think DefCore actually needs to change old versions of Tempest,
> > but maybe Chris or Mark can verify that?
> 
> So if I’m groking this correctly, there’s kind of two scenarios
> being painted here.  One is the “LCD” approach where we use the
> $osversion-eol version of Tempest, where $osversion matches the
> oldest version covered in a Guideline.  The other is to use the
> start-of-$osversion version of Tempest where $osversion is the
> OpenStack version after the most recent one in the Guideline.  The
> former may result in some fairly long-lived flags, and the latter
> is actually not terribly different than what we do today I think.
> Let me try to talk through both...
> 
> In some cases, tests get flagged in the Guidelines because of
> bugs in the test or because the test needs refactoring.  The
> underlying Capabilities the those tests are testing actually work
> fine.  Once we identify such an issue, the test can be fixed…in
> master.  Under the first scenario, this potentially creates some
> very long-lived flags:
> 
> 2016.01 is the most current Guideline right now covers Juno, Kilo,
> Liberty (and Mitaka after it was released).  It’s one of the two
> Guidelines that you can use if you want an OpenStack Powered license
> from he Foundation, $vendor wants to run it against their shiny new
> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
> they find a test issue, and we flag it.  A few weeks later, a fix
> lands in Tempest.  Several months later the next Guideline rolls
> around: the oldest covered release is Kilo and we start telling
> people to use the Kilo-EOL version of Tempest.  That doesn’t have
> the fix, so the flag stays.  Another six months goes by and we get
> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
> that's the first version that includes the fix.
>
> Generally speaking long lived flags aren’t so great because it
> means the tests are not required…which means there’s less or no
> assurance that the capabilities they test for actually work in the
> clouds that adhere to those Guidelines.  So, the worst-case scenario
> here looks kind of ugly.
> 
> As Matt correctly pointed out though, the capabilities DefCore
> selects for are generally pretty stable API’s that are long-lived
> across many releases, so we haven’t run into a lot of issues running
> pretty new versions of Tempest against older clouds to date.  In
> fact I’m struggling to think of a time we’ve flagged something
> because someone complained the test wasn’t runnable against an older
> release covered by the Guideline in question.  I can think of plenty
> of times where we’ve flagged something due to a test issue though…keep
> in mind we’re still in pretty formative times with DefCore here
> where these tests are starting to be used in a new way for the first
> time.  Anyway, as Matt points out we could potentially use a much
> newer Tempest tag: tag=11 (which is the start of Newton development
> and is a roughly 2 month old version of Tempest).  Next Guideline
> rolls around, we use the tag for start-of-ocata, and we get the fix
> and can drop the flag.
> 
> Today, RefStack client by default checks out a specific SHA of
> Tempest [1] (it actually did use a tag at some point in the past,
> and still can).  When we see a fix for a flagged test go in, we or
> the Refstack folks can do a quick test to make sure everything’s
> in order and then update that SHA to match the version with the
> fix.  That way we’re relatively sure we have a version that works
> today, and will work when we drop the flag in the next Guideline
> too.  When we finalize that next Guideline, we also update the
> test-repositories section of the new Guideline that Matt pointed
> to earlier to reflect the best-known version on the day the Guideline
> was sent to the Board for approval.  One added benefit of this
> approach is that people running the tests today may get a version
> of Tempest that includes a fix for a flagged test.  A flagged test
> isn’t required, but it does get run—and now will show a passing
> result, so we have data that says “this provider actually does
> support this capability (even though it’s flagged), and the test
> does indeed seem to be working."
> 
> 
> So, that’s actually not hugely different from the second scenario
> I think?  Or did I miss something there?

What I was proposing is that we keep the certification rules in
sync with OpenStack versions. If a vendor stays on an old version
of OpenStack, they can keep using the old (matching) version of
Tempest to certify.  It's not clear from your description if that's
allowed now.

We would want to expose the version of OpenStack, the version of
Tempest, any flagged tests, and any configuration settings that
change 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/14/2016 07:19 PM, Chris Hoge wrote:
> 
>> On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
>>
>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
>>
>>> But, if we add another possible state on the defcore side like conditional 
>>> pass,
>>> warning, yellow, etc. (the name doesn't matter) which is used to indicate 
>>> that
>>> things on product X could only pass when strict validation was disabled (and
>>> be clear about where and why) then my concerns would be alleviated. I just 
>>> do
>>> not want this to end up not being visible to end users trying to evaluate
>>> interoperability of different clouds using the test results.
>>
>> +1
>>
>> Don't fail them, but don't cover up their incompatibility, either.
>> -- Ed Leafe
> 
> That’s not my proposal. My requirement is that vendors who want to do this
> state exactly which APIs are sending back additional data, and that this
> information be published.
> 
> There are different levels of incompatibility. A response with additional data
> that can be safely ignored is different from a changed response that would
> cause a client to fail.

It's actually not different. It's really not.

This idea that it's safe to add response data is based on an assumption
that software versions only move forward. If you have a single deploy of
software, that's fine.

However as noted, we've got production clouds on Juno <-> Mitaka in the
wild. Which means if we want to support horizontal transfer between
clouds, the user experienced timeline might be start on a Mitaka cloud,
then try to move to Juno. So anything added from Juno -> Mitaka without
signaling has exactly the same client breaking behavior as removing
attributes.

Which is why microversions are needed for attribute adds.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Ken'ichi Ohmichi]

2016-06-16 2:26 GMT-07:00 Morgan Fainberg :
> On Wed, Jun 15, 2016 at 11:54 PM, Ken'ichi Ohmichi 
> wrote:
>>
>> This discussion was expected when we implemented the Tempest patch,
>> then I sent a mail to defcore comittee[1]
>> As the above ml, "A DefCore Guideline typically covers three OpenStack
>> releases".
>> That means the latest guideline needs to cover Mitaka, Liberty and Kilo,
>> right?
>>
>> In the Kilo development, we(nova team) have already considered
>> additional properties are not good for the interoperability.
>> And the stable_api.rst of [2] which is contained in Kilo says we need
>> to implement new features without extensions.
>> However, there are Kilo+ clouds which are extended with vendors' own
>> extensions, right?
>>
>> My concern of allowing additional properties on interoperability tests is
>> that
>>  - users can move from pure OpenStack clouds to non-pure OpenStack
>> clouds which implement vender specific properties
>>  - but users cannot move from non-pure OpenStack clouds if users
>> depend on the properties
>> even if these clouds are certificated on the same interoperability tests.
>>
>
> The end goal is 100% to get everyone consistent with no "extra" data being
> passed out of the APIs and certified on the same tests.

Yeah, I am appreciated that everyone agree with the non-extra data as
the final goal.

> However, right now we have an issue where vendors/operators are lagging on
> getting this cleaned up. Since this is the first round of certifications
> (among other things), the proposal is to support/manage this in a way that
> gives a bit more of a grace period while the deployers/operators finish
> moving away from custom properties (as i understand it the ones affected
> have communicated that they are working on meeting this goal; Chris, please
> correct me if I am wrong).
>
> Your concerns are spot on, and at the end of this "greylist" window ( at the
> " 2017.01" defcore guideline ), this grace period will expire and everyone
> will be expected to be compatible without the "Extra" data. Part of the
> process of doing these programs is working to refine the process (and
> sometimes make exceptions in the early stages) until the workflow is
> established and understood. It is not expected to continue nor extend the
> period beyond the firm end point Chris highlighted. I would not support this
> proposal if it was open ended.

The greylist seems a good idea, and I am not so strongly against the idea.
However, I have still some questions about this direction.

I am thinking most important API of Nova is "create a server" API for
the interoperability, because most users want to use servers on
OpenStack clouds.
However, I am guessing most venders which cannot be passed through
current strict Tempest are customizing this API.
So if this API on the greylist on most venders' tests, the
interoperability seems a little meaningless.
Is that expected now?

One more question is that how many venders cannot pass through current Tempest?
100%? or 20%?
If 5% venders cannot pass, I guess we can say "the certification is
failed" to the venders.
I'd like to know current situation for expecting our future so that we
will need to mark this "greylist" as deprecated soon and need to know
how progress at some steps/cycles of venders.

Thanks
Ken Omichi

---
>> ---
>> [1]:
>> http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
>> [2]: https://review.openstack.org/#/c/162912
>>
>> 2016-06-14 16:37 GMT-07:00 Chris Hoge :
>> > Top posting one note and direct comments inline, I’m proposing
>> > this as a member of the DefCore working group, but this
>> > proposal itself has not been accepted as the forward course of
>> > action by the working group. These are my own views as the
>> > administrator of the program and not that of the working group
>> > itself, which may independently reject the idea outside of the
>> > response from the upstream devs.
>> >
>> > I posted a link to this thread to the DefCore mailing list to make
>> > that working group aware of the outstanding issues.
>> >
>> > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
>> > wrote:
>> >
>> > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> >
>> > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>> >
>> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>> >
>> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
>> >
>> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> >
>> > Last year, in response to Nova micro-versioning and extension
>> > updates[1],
>> > the QA team added strict API schema checking to Tempest to ensure that
>> > no additional properties were added to Nova API responses[2][3]. In the
>> > last year, at least three vendors participating the the OpenStack
>> > Powered
>> > Trademark program have been 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matt Riedemann]

On 6/14/2016 6:37 PM, Chris Hoge wrote:

I’m beginning to wonder if we need to make DefCore use release
branches then back-port bug-fixes and relevant features additions
as necessary.



To clarify, do you mean use release branches from the upstream repos? 
Because if DefCore is supporting 2 years of releases, that's going to 
conflict with the 18 month lifespan of branches upstream, i.e. kilo is 
already EOL. So backporting to stable/juno or stable/kilo at this point 
isn't possible, at least in the official openstack repos.


And backporting features to stable branches would break our stable 
branch policy.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matt Riedemann]

On 6/14/2016 6:19 PM, Chris Hoge wrote:

One would hope that micro-versions would be able to address this exact
issue for vendors by giving them a means to propose optional but
well-defined API response additions (not extensions) that are defined
upstream and usable by all vendors. If it’s not too off topic, can someone
from the Nova team explain how something like that would work (if it
would at all)?

-Chris
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



The 2.4 microversion is about as simple as it gets to adding more data 
to a response:


http://docs.openstack.org/developer/nova/api_microversion_history.html#id3

If you're requesting microversion >=2.4 you get the 'reserved' flag back 
when showing details on a fixed IP, else it's omitted from the response 
(as in v2.0 and v2.1).


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mike Perez]

On 23:53 Jun 17, Matthew Treinish wrote:
> On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote:
> > On 15:12 Jun 14, Matthew Treinish wrote:



> > > I don't think backwards compatibility policies really apply to what what 
> > > define
> > > as the set of tests that as a community we are saying a vendor has to 
> > > pass to
> > > say they're OpenStack. From my perspective as a community we either take 
> > > a hard
> > > stance on this and say to be considered an interoperable cloud (and to 
> > > get the
> > > trademark) you have to actually have an interoperable product. We slowly 
> > > ratchet
> > > up the requirements every 6 months, there isn't any implied backwards
> > > compatibility in doing that. You passed in the past but not in the newer 
> > > stricter
> > > guidelines.
> > > 
> > > Also, even if I did think it applied, we're not talking about a change 
> > > which
> > > would fall into breaking that. The change was introduced a year and half 
> > > ago
> > > during kilo and landed a year ago during liberty:
> > > 
> > > https://review.openstack.org/#/c/156130/
> > > 
> > > That's way longer than our normal deprecation period of 3 months and a 
> > > release
> > > boundary.
> > 
> > 
> > 
> > What kind of communication happens today for these changes? There are so 
> > many
> > channels/high volume mailing lists a downstream deployer is expected by the
> > community to listening in. Some disruptive change being introduced a year or
> > longer ago can still be communicated poorly.
> 
> Sure, I agree with that, but I don't think this was necessarily communicated
> poorly. This has been already mentioned a few times on this thread but:
> 
> It was talked about on openstack-dev:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
> 
> On the defcore list: (which is definitely not high volume/traffic ML)
> 
> http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
> 
> This was also raised as an issue for 1 vendor ~6 months ago. (which is also 
> the
> same duration of the hard deadline being discussed in this thread):
> 
> http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
> 
> IMHO, this was more than enough time to introduce a fix or workaround on their
> end. Likely the easiest being just adding an extra nova-api endpoint with the
> extensions disabled.
> 
> I don't have any links or other evidence to point to, but I know that this
> exact topic has been discussed with with people from the vendors having
> difficulties during sessions at at least one of the 2 summits and/or 2 QA
> midcycle meetups since this change landed. I really don't think this is a
> communication problem or unfair surprise for anyone.
> 
> There might be more too, but I don't remember every conversation that I've had
> in the community over the past year. (or where to find the links to point to)

Thanks for the references. So these references show:

* DefCore was aware of these changes long ago.
* Vendors were aware of these changes long ago.
* Referenced vendor is still failing after knowing about this change for six
  months.

Question to DefCore, what were you doing in this time frame to prepare vendors
for success with this change rolling out in order to keep a healthy market
place?

> > Just like we've done with Reno in communicating better about disruptive 
> > changes
> > in release notes, what tells teams like DefCore about changes with Tempest?
> > (I looked in release.o.o for tempest release notes, although maybe I missed
> > it?)
> 
> Yes, tempest has release notes, they are here:
> 
> http://docs.openstack.org/releasenotes/tempest/
> 
> But, the change in question predates the existence of reno and centralized
> release notes for everything in openstack.
> 
> If this change were pushed today it would definitely be included in the 
> release
> notes. We also would do the same things, put it on the dev list, put it on the
> defcore list. (although probably as a standalone thread this time) I also 
> think
> we'd probably ping hogepodge on irc about it too just so he could also raise 
> it
> up on the defcore side. (which we might have done back then too) Defcore and
> tempest are tightly coupled so we do have pretty constant communication around
> changes being made. But, I do admit we have better mechanisms in place today
> to communicate this kind of change, and hopefully this would be handled better
> now.

This is great! I hope people who have use cases with Tempest are using these
release notes for future big changes.

> > 
> > Since some members of DefCore have interest in making the market place 
> > healthy,
> > what is DefCore doing today to communicate these disruptive changes early to
> > deployers? Did it not happen in this particular case because:
> > 
> > * DefCore has no one working closely in the Tempest project to flag things?
> > * Defcore does work closely with Tempest, but somehow the communication for
> 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote:
> On 15:12 Jun 14, Matthew Treinish wrote:
> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> 
> 
> 
> > > We have basically three options.
> > > 
> > > 1. Tell deployers who are trying to do the right for their immediate
> > >users that they can't use the trademark.
> > > 
> > > 2. Flag the related tests or remove them from the DefCore enforcement
> > >suite entirely.
> > > 
> > > 3. Be flexible about giving consumers of Tempest time to meet the
> > >new requirement by providing a way to disable the checks.
> > > 
> > > Option 1 goes against our own backwards compatibility policies.
> > 
> > I don't think backwards compatibility policies really apply to what what 
> > define
> > as the set of tests that as a community we are saying a vendor has to pass 
> > to
> > say they're OpenStack. From my perspective as a community we either take a 
> > hard
> > stance on this and say to be considered an interoperable cloud (and to get 
> > the
> > trademark) you have to actually have an interoperable product. We slowly 
> > ratchet
> > up the requirements every 6 months, there isn't any implied backwards
> > compatibility in doing that. You passed in the past but not in the newer 
> > stricter
> > guidelines.
> > 
> > Also, even if I did think it applied, we're not talking about a change which
> > would fall into breaking that. The change was introduced a year and half ago
> > during kilo and landed a year ago during liberty:
> > 
> > https://review.openstack.org/#/c/156130/
> > 
> > That's way longer than our normal deprecation period of 3 months and a 
> > release
> > boundary.
> 
> 
> 
> What kind of communication happens today for these changes? There are so many
> channels/high volume mailing lists a downstream deployer is expected by the
> community to listening in. Some disruptive change being introduced a year or
> longer ago can still be communicated poorly.

Sure, I agree with that, but I don't think this was necessarily communicated
poorly. This has been already mentioned a few times on this thread but:

It was talked about on openstack-dev:

http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html

On the defcore list: (which is definitely not high volume/traffic ML)

http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html

This was also raised as an issue for 1 vendor ~6 months ago. (which is also the
same duration of the hard deadline being discussed in this thread):

http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html

IMHO, this was more than enough time to introduce a fix or workaround on their
end. Likely the easiest being just adding an extra nova-api endpoint with the
extensions disabled.

I don't have any links or other evidence to point to, but I know that this
exact topic has been discussed with with people from the vendors having
difficulties during sessions at at least one of the 2 summits and/or 2 QA
midcycle meetups since this change landed. I really don't think this is a
communication problem or unfair surprise for anyone.

There might be more too, but I don't remember every conversation that I've had
in the community over the past year. (or where to find the links to point to)

> 
> Just like we've done with Reno in communicating better about disruptive 
> changes
> in release notes, what tells teams like DefCore about changes with Tempest?
> (I looked in release.o.o for tempest release notes, although maybe I missed
> it?)

Yes, tempest has release notes, they are here:

http://docs.openstack.org/releasenotes/tempest/

But, the change in question predates the existence of reno and centralized
release notes for everything in openstack.

If this change were pushed today it would definitely be included in the release
notes. We also would do the same things, put it on the dev list, put it on the
defcore list. (although probably as a standalone thread this time) I also think
we'd probably ping hogepodge on irc about it too just so he could also raise it
up on the defcore side. (which we might have done back then too) Defcore and
tempest are tightly coupled so we do have pretty constant communication around
changes being made. But, I do admit we have better mechanisms in place today
to communicate this kind of change, and hopefully this would be handled better
now.

> 
> Since some members of DefCore have interest in making the market place 
> healthy,
> what is DefCore doing today to communicate these disruptive changes early to
> deployers? Did it not happen in this particular case because:
> 
> * DefCore has no one working closely in the Tempest project to flag things?
> * Defcore does work closely with Tempest, but somehow the communication for
>   this was missed?
> * Not having clear 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mike Perez]

On 15:12 Jun 14, Matthew Treinish wrote:
> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:



> > We have basically three options.
> > 
> > 1. Tell deployers who are trying to do the right for their immediate
> >users that they can't use the trademark.
> > 
> > 2. Flag the related tests or remove them from the DefCore enforcement
> >suite entirely.
> > 
> > 3. Be flexible about giving consumers of Tempest time to meet the
> >new requirement by providing a way to disable the checks.
> > 
> > Option 1 goes against our own backwards compatibility policies.
> 
> I don't think backwards compatibility policies really apply to what what 
> define
> as the set of tests that as a community we are saying a vendor has to pass to
> say they're OpenStack. From my perspective as a community we either take a 
> hard
> stance on this and say to be considered an interoperable cloud (and to get the
> trademark) you have to actually have an interoperable product. We slowly 
> ratchet
> up the requirements every 6 months, there isn't any implied backwards
> compatibility in doing that. You passed in the past but not in the newer 
> stricter
> guidelines.
> 
> Also, even if I did think it applied, we're not talking about a change which
> would fall into breaking that. The change was introduced a year and half ago
> during kilo and landed a year ago during liberty:
> 
> https://review.openstack.org/#/c/156130/
> 
> That's way longer than our normal deprecation period of 3 months and a release
> boundary.



What kind of communication happens today for these changes? There are so many
channels/high volume mailing lists a downstream deployer is expected by the
community to listening in. Some disruptive change being introduced a year or
longer ago can still be communicated poorly.

Just like we've done with Reno in communicating better about disruptive changes
in release notes, what tells teams like DefCore about changes with Tempest?
(I looked in release.o.o for tempest release notes, although maybe I missed
it?)

Since some members of DefCore have interest in making the market place healthy,
what is DefCore doing today to communicate these disruptive changes early to
deployers? Did it not happen in this particular case because:

* DefCore has no one working closely in the Tempest project to flag things?
* Defcore does work closely with Tempest, but somehow the communication for
  this was missed?
* Not having clear deprecation notices because release notes in the Tempest
  don't exist (see above)?

This all just sounds like a communication problem, and it makes me sad to
interpret this thread as people being angry with deployers as a result. How
about we not think the worse of people that are trying to prove our project
being successful and start working with them?

With that said I agree with this strict checking in tests. Deployments need to
stop defining the community defined APIs.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[John Garbutt]

On 15 June 2016 at 02:37, Sean Dague  wrote:
> On 06/14/2016 07:28 PM, Monty Taylor wrote:
>>
>> On 06/14/2016 05:42 PM, Doug Hellmann wrote:
>
> 
>>
>> I think this is the most important thing to me as it relates to this.
>> I'm obviously a huge proponent of clouds behaving more samely. But I
>> also think that, as Doug nicely describes above, we've sort of backed in
>> to removing something without a deprecation window ... largely because
>> of the complexities involved with the system here - and I'd like to make
>> sure that when we are being clear about behavior changes that we give
>> the warning period so that people can adapt.

+1

> I also think that "pass" to "pass *"  is useful social incentive. While I
> think communication of this new direction has happened pretty broadly,
> organizations are complex places, and it didn't filter everywhere it needed
> to with the urgency that was probably needed.
>
> "pass *"  * - with a greylist which goes away in 6 months
>
> Will hopefully be a reasonable enough push to get the behavior we want,
> which is everyone publishing the same interface.

+1

I know I am going back in time, but +1 all the same.

I like how this pushes forward the of interoperability cause.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



On Jun 16, 2016, at 2:25 PM, Matthew Treinish  wrote:

On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
Top posting one note and direct comments inline, I’m proposing
this as a member of the DefCore working group, but this
proposal itself has not been accepted as the forward course of
action by the working group. These are my own views as the
administrator of the program and not that of the working group
itself, which may independently reject the idea outside of the
response from the upstream devs.

I posted a link to this thread to the DefCore mailing list to make
that working group aware of the outstanding issues.

On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:

On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
Last year, in response to Nova micro-versioning and extension updates[1],
the QA team added strict API schema checking to Tempest to ensure that
no additional properties were added to Nova API responses[2][3]. In the
last year, at least three vendors participating the the OpenStack Powered
Trademark program have been impacted by this change, two of which
reported this to the DefCore Working Group mailing list earlier this year[4].

The DefCore Working Group determines guidelines for the OpenStack Powered
program, which includes capabilities with associated functional tests
from Tempest that must be passed, and designated sections with associated
upstream code [5][6]. In determining these guidelines, the working group
attempts to balance the future direction of development with lagging
indicators of deployments and user adoption.

After a tremendous amount of consideration, I believe that the DefCore
Working Group needs to implement a temporary waiver for the strict API
checking requirements that were introduced last year, to give downstream
deployers more time to catch up with the strict micro-versioning
requirements determined by the Nova/Compute team and enforced by the
Tempest/QA team.

I'm very much opposed to this being done. If we're actually concerned with
interoperability and verify that things behave in the same manner between 
multiple
clouds then doing this would be a big step backwards. The fundamental disconnect
here is that the vendors who have implemented out of band extensions or were
taking advantage of previously available places to inject extra attributes
believe that doing so means they're interoperable, which is quite far from
reality. **The API is not a place for vendor differentiation.**

This is a temporary measure to address the fact that a large number
of existing tests changed their behavior, rather than having new
tests added to enforce this new requirement. The result is deployments
that previously passed these tests may no longer pass, and in fact
we have several cases where that's true with deployers who are
trying to maintain their own standard of backwards-compatibility
for their end users.

That's not what happened though. The API hasn't changed and the tests haven't
really changed either. We made our enforcement on Nova's APIs a bit stricter to
ensure nothing unexpected appeared. For the most these tests work on any version
of OpenStack. (we only test it in the gate on supported stable releases, but I
don't expect things to have drastically shifted on older releases) It also
doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the
only case it ever fails is when you run something extra, not from the community,
either as an extension (which themselves are going away [1]) or another service
that wraps nova or imitates nova. I'm personally not comfortable saying those
extras are ever part of the OpenStack APIs.

We have basically three options.

1. Tell deployers who are trying to do the right for their immediate
 users that they can't use the trademark.

2. Flag the related tests or remove them from the DefCore enforcement
 suite entirely.

3. Be flexible about giving consumers of Tempest time to meet the
 new requirement by providing a way to disable the checks.

Option 1 goes against our own backwards compatibility policies.

I don't think backwards compatibility policies really apply to what what define
as the set of tests that as a community we are saying a vendor has to pass to
say they're OpenStack. From my perspective as a 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
> > On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > > > Top posting one note and direct comments inline, I’m proposing
> > > > > > this as a member of the DefCore working group, but this
> > > > > > proposal itself has not been accepted as the forward course of
> > > > > > action by the working group. These are my own views as the
> > > > > > administrator of the program and not that of the working group
> > > > > > itself, which may independently reject the idea outside of the
> > > > > > response from the upstream devs.
> > > > > > 
> > > > > > I posted a link to this thread to the DefCore mailing list to make
> > > > > > that working group aware of the outstanding issues.
> > > > > > 
> > > > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
> > > > > > >  wrote:
> > > > > > > 
> > > > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > > > >> -0400:
> > > > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > > > >  Excerpts from Matthew Treinish's message of 2016-06-14 
> > > > > >  14:21:27 -0400:
> > > > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > > >> Last year, in response to Nova micro-versioning and 
> > > > > > >> extension updates[1],
> > > > > > >> the QA team added strict API schema checking to Tempest to 
> > > > > > >> ensure that
> > > > > > >> no additional properties were added to Nova API 
> > > > > > >> responses[2][3]. In the
> > > > > > >> last year, at least three vendors participating the the 
> > > > > > >> OpenStack Powered
> > > > > > >> Trademark program have been impacted by this change, two of 
> > > > > > >> which
> > > > > > >> reported this to the DefCore Working Group mailing list 
> > > > > > >> earlier this year[4].
> > > > > > >> 
> > > > > > >> The DefCore Working Group determines guidelines for the 
> > > > > > >> OpenStack Powered
> > > > > > >> program, which includes capabilities with associated 
> > > > > > >> functional tests
> > > > > > >> from Tempest that must be passed, and designated sections 
> > > > > > >> with associated
> > > > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > > > >> working group
> > > > > > >> attempts to balance the future direction of development with 
> > > > > > >> lagging
> > > > > > >> indicators of deployments and user adoption.
> > > > > > >> 
> > > > > > >> After a tremendous amount of consideration, I believe that 
> > > > > > >> the DefCore
> > > > > > >> Working Group needs to implement a temporary waiver for the 
> > > > > > >> strict API
> > > > > > >> checking requirements that were introduced last year, to 
> > > > > > >> give downstream
> > > > > > >> deployers more time to catch up with the strict 
> > > > > > >> micro-versioning
> > > > > > >> requirements determined by the Nova/Compute team and 
> > > > > > >> enforced by the
> > > > > > >> Tempest/QA team.
> > > > > > > 
> > > > > > > I'm very much opposed to this being done. If we're actually 
> > > > > > > concerned with
> > > > > > > interoperability and verify that things behave in the same 
> > > > > > > manner between multiple
> > > > > > > clouds then doing this would be a big step backwards. The 
> > > > > > > fundamental disconnect
> > > > > > > here is that the vendors who have implemented out of band 
> > > > > > > extensions or were
> > > > > > > taking advantage of previously available places to inject 
> > > > > > > extra attributes
> > > > > > > believe that doing so means they're interoperable, which is 
> > > > > > > quite far from
> > > > > > > reality. **The API is not a place for vendor 
> > > > > > > differentiation.**
> > > > > >  
> > > > > >  This is a temporary measure to address the fact that a large 
> > > > > >  number
> > > > > >  of existing tests changed their behavior, rather than having 
> > > > > >  new
> > > > > >  tests added to enforce this new requirement. The result is 
> > > > > >  deployments
> > > > > >  that previously passed these tests may no longer pass, and in 
> > > > > >  fact
> > > > > >  we have several cases where that's true with deployers who are
> > > > > >  trying to maintain their own standard of 
> > > > > >  backwards-compatibility
> > > > > >  for their end 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
> On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > > Top posting one note and direct comments inline, I’m proposing
> > > > > this as a member of the DefCore working group, but this
> > > > > proposal itself has not been accepted as the forward course of
> > > > > action by the working group. These are my own views as the
> > > > > administrator of the program and not that of the working group
> > > > > itself, which may independently reject the idea outside of the
> > > > > response from the upstream devs.
> > > > > 
> > > > > I posted a link to this thread to the DefCore mailing list to make
> > > > > that working group aware of the outstanding issues.
> > > > > 
> > > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
> > > > > >  wrote:
> > > > > > 
> > > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > > >> -0400:
> > > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > > > >  -0400:
> > > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > >> Last year, in response to Nova micro-versioning and extension 
> > > > > >> updates[1],
> > > > > >> the QA team added strict API schema checking to Tempest to 
> > > > > >> ensure that
> > > > > >> no additional properties were added to Nova API 
> > > > > >> responses[2][3]. In the
> > > > > >> last year, at least three vendors participating the the 
> > > > > >> OpenStack Powered
> > > > > >> Trademark program have been impacted by this change, two of 
> > > > > >> which
> > > > > >> reported this to the DefCore Working Group mailing list 
> > > > > >> earlier this year[4].
> > > > > >> 
> > > > > >> The DefCore Working Group determines guidelines for the 
> > > > > >> OpenStack Powered
> > > > > >> program, which includes capabilities with associated 
> > > > > >> functional tests
> > > > > >> from Tempest that must be passed, and designated sections with 
> > > > > >> associated
> > > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > > >> working group
> > > > > >> attempts to balance the future direction of development with 
> > > > > >> lagging
> > > > > >> indicators of deployments and user adoption.
> > > > > >> 
> > > > > >> After a tremendous amount of consideration, I believe that the 
> > > > > >> DefCore
> > > > > >> Working Group needs to implement a temporary waiver for the 
> > > > > >> strict API
> > > > > >> checking requirements that were introduced last year, to give 
> > > > > >> downstream
> > > > > >> deployers more time to catch up with the strict 
> > > > > >> micro-versioning
> > > > > >> requirements determined by the Nova/Compute team and enforced 
> > > > > >> by the
> > > > > >> Tempest/QA team.
> > > > > > 
> > > > > > I'm very much opposed to this being done. If we're actually 
> > > > > > concerned with
> > > > > > interoperability and verify that things behave in the same 
> > > > > > manner between multiple
> > > > > > clouds then doing this would be a big step backwards. The 
> > > > > > fundamental disconnect
> > > > > > here is that the vendors who have implemented out of band 
> > > > > > extensions or were
> > > > > > taking advantage of previously available places to inject extra 
> > > > > > attributes
> > > > > > believe that doing so means they're interoperable, which is 
> > > > > > quite far from
> > > > > > reality. **The API is not a place for vendor differentiation.**
> > > > >  
> > > > >  This is a temporary measure to address the fact that a large 
> > > > >  number
> > > > >  of existing tests changed their behavior, rather than having new
> > > > >  tests added to enforce this new requirement. The result is 
> > > > >  deployments
> > > > >  that previously passed these tests may no longer pass, and in 
> > > > >  fact
> > > > >  we have several cases where that's true with deployers who are
> > > > >  trying to maintain their own standard of backwards-compatibility
> > > > >  for their end users.
> > > > > >>> 
> > > > > >>> That's not what happened though. The API hasn't changed and the 
> > > > > >>> tests haven't
> > > > > >>> really changed either. We made our enforcement on Nova's APIs a 
> > > > > >>> bit stricter to
> > > > > >>> ensure nothing unexpected appeared. For 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > Top posting one note and direct comments inline, I’m proposing
> > > > this as a member of the DefCore working group, but this
> > > > proposal itself has not been accepted as the forward course of
> > > > action by the working group. These are my own views as the
> > > > administrator of the program and not that of the working group
> > > > itself, which may independently reject the idea outside of the
> > > > response from the upstream devs.
> > > > 
> > > > I posted a link to this thread to the DefCore mailing list to make
> > > > that working group aware of the outstanding issues.
> > > > 
> > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  
> > > > > wrote:
> > > > > 
> > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > >> -0400:
> > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > > >  -0400:
> > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > >> Last year, in response to Nova micro-versioning and extension 
> > > > >> updates[1],
> > > > >> the QA team added strict API schema checking to Tempest to 
> > > > >> ensure that
> > > > >> no additional properties were added to Nova API responses[2][3]. 
> > > > >> In the
> > > > >> last year, at least three vendors participating the the 
> > > > >> OpenStack Powered
> > > > >> Trademark program have been impacted by this change, two of which
> > > > >> reported this to the DefCore Working Group mailing list earlier 
> > > > >> this year[4].
> > > > >> 
> > > > >> The DefCore Working Group determines guidelines for the 
> > > > >> OpenStack Powered
> > > > >> program, which includes capabilities with associated functional 
> > > > >> tests
> > > > >> from Tempest that must be passed, and designated sections with 
> > > > >> associated
> > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > >> working group
> > > > >> attempts to balance the future direction of development with 
> > > > >> lagging
> > > > >> indicators of deployments and user adoption.
> > > > >> 
> > > > >> After a tremendous amount of consideration, I believe that the 
> > > > >> DefCore
> > > > >> Working Group needs to implement a temporary waiver for the 
> > > > >> strict API
> > > > >> checking requirements that were introduced last year, to give 
> > > > >> downstream
> > > > >> deployers more time to catch up with the strict micro-versioning
> > > > >> requirements determined by the Nova/Compute team and enforced by 
> > > > >> the
> > > > >> Tempest/QA team.
> > > > > 
> > > > > I'm very much opposed to this being done. If we're actually 
> > > > > concerned with
> > > > > interoperability and verify that things behave in the same manner 
> > > > > between multiple
> > > > > clouds then doing this would be a big step backwards. The 
> > > > > fundamental disconnect
> > > > > here is that the vendors who have implemented out of band 
> > > > > extensions or were
> > > > > taking advantage of previously available places to inject extra 
> > > > > attributes
> > > > > believe that doing so means they're interoperable, which is quite 
> > > > > far from
> > > > > reality. **The API is not a place for vendor differentiation.**
> > > >  
> > > >  This is a temporary measure to address the fact that a large number
> > > >  of existing tests changed their behavior, rather than having new
> > > >  tests added to enforce this new requirement. The result is 
> > > >  deployments
> > > >  that previously passed these tests may no longer pass, and in fact
> > > >  we have several cases where that's true with deployers who are
> > > >  trying to maintain their own standard of backwards-compatibility
> > > >  for their end users.
> > > > >>> 
> > > > >>> That's not what happened though. The API hasn't changed and the 
> > > > >>> tests haven't
> > > > >>> really changed either. We made our enforcement on Nova's APIs a bit 
> > > > >>> stricter to
> > > > >>> ensure nothing unexpected appeared. For the most these tests work 
> > > > >>> on any version
> > > > >>> of OpenStack. (we only test it in the gate on supported stable 
> > > > >>> releases, but I
> > > > >>> don't expect things to have drastically shifted on older releases) 
> > > > >>> It also
> > > > >>> doesn't matter which version of the API you 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > Top posting one note and direct comments inline, I’m proposing
> > > this as a member of the DefCore working group, but this
> > > proposal itself has not been accepted as the forward course of
> > > action by the working group. These are my own views as the
> > > administrator of the program and not that of the working group
> > > itself, which may independently reject the idea outside of the
> > > response from the upstream devs.
> > > 
> > > I posted a link to this thread to the DefCore mailing list to make
> > > that working group aware of the outstanding issues.
> > > 
> > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  
> > > > wrote:
> > > > 
> > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > >  -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > >> Last year, in response to Nova micro-versioning and extension 
> > > >> updates[1],
> > > >> the QA team added strict API schema checking to Tempest to ensure 
> > > >> that
> > > >> no additional properties were added to Nova API responses[2][3]. 
> > > >> In the
> > > >> last year, at least three vendors participating the the OpenStack 
> > > >> Powered
> > > >> Trademark program have been impacted by this change, two of which
> > > >> reported this to the DefCore Working Group mailing list earlier 
> > > >> this year[4].
> > > >> 
> > > >> The DefCore Working Group determines guidelines for the OpenStack 
> > > >> Powered
> > > >> program, which includes capabilities with associated functional 
> > > >> tests
> > > >> from Tempest that must be passed, and designated sections with 
> > > >> associated
> > > >> upstream code [5][6]. In determining these guidelines, the working 
> > > >> group
> > > >> attempts to balance the future direction of development with 
> > > >> lagging
> > > >> indicators of deployments and user adoption.
> > > >> 
> > > >> After a tremendous amount of consideration, I believe that the 
> > > >> DefCore
> > > >> Working Group needs to implement a temporary waiver for the strict 
> > > >> API
> > > >> checking requirements that were introduced last year, to give 
> > > >> downstream
> > > >> deployers more time to catch up with the strict micro-versioning
> > > >> requirements determined by the Nova/Compute team and enforced by 
> > > >> the
> > > >> Tempest/QA team.
> > > > 
> > > > I'm very much opposed to this being done. If we're actually 
> > > > concerned with
> > > > interoperability and verify that things behave in the same manner 
> > > > between multiple
> > > > clouds then doing this would be a big step backwards. The 
> > > > fundamental disconnect
> > > > here is that the vendors who have implemented out of band 
> > > > extensions or were
> > > > taking advantage of previously available places to inject extra 
> > > > attributes
> > > > believe that doing so means they're interoperable, which is quite 
> > > > far from
> > > > reality. **The API is not a place for vendor differentiation.**
> > >  
> > >  This is a temporary measure to address the fact that a large number
> > >  of existing tests changed their behavior, rather than having new
> > >  tests added to enforce this new requirement. The result is 
> > >  deployments
> > >  that previously passed these tests may no longer pass, and in fact
> > >  we have several cases where that's true with deployers who are
> > >  trying to maintain their own standard of backwards-compatibility
> > >  for their end users.
> > > >>> 
> > > >>> That's not what happened though. The API hasn't changed and the tests 
> > > >>> haven't
> > > >>> really changed either. We made our enforcement on Nova's APIs a bit 
> > > >>> stricter to
> > > >>> ensure nothing unexpected appeared. For the most these tests work on 
> > > >>> any version
> > > >>> of OpenStack. (we only test it in the gate on supported stable 
> > > >>> releases, but I
> > > >>> don't expect things to have drastically shifted on older releases) It 
> > > >>> also
> > > >>> doesn't matter which version of the API you run, v2.0 or v2.1. 
> > > >>> Literally, the
> > > >>> only case it ever fails is when you run something extra, not from the 
> > > >>> community,
> > > >>> either as an extension (which themselves are going away [1]) or 
> > > >>> another service
> > > >>> 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Hayes, Graham]

On 16/06/2016 00:30, Matthew Treinish wrote:
> On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
>> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
>>> Top posting one note and direct comments inline, I’m proposing
>>> this as a member of the DefCore working group, but this
>>> proposal itself has not been accepted as the forward course of
>>> action by the working group. These are my own views as the
>>> administrator of the program and not that of the working group
>>> itself, which may independently reject the idea outside of the
>>> response from the upstream devs.
>>>
>>> I posted a link to this thread to the DefCore mailing list to make
>>> that working group aware of the outstanding issues.
>>>
 On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:

 On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
 On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:



>>> The current active guidelines cover icehouse through mitaka. The release
>>> of 2016.08 will change that to cover juno through mitaka (with newton
>>> as an add-on to 2016.08 when it’s released). There’s overlap between
>>> the guidelines, so 2016.01 covers juno through mitaka while 2016.08
>>> will cover kilo through newton. Essentially two years of releases.
>>>
> We may also need to consider that test implementation details may
> change, and have a review process within DefCore to help expose
> those changes to make them clearer to deployers.
>
> Fixing the process issue may also mean changing the way we implement
> things in Tempest. In this case, adding a flag helps move ahead
> more smoothly. Perhaps we adopt that as a general policy in the
> future when we make underlying behavioral changes like this to
> existing tests.  Perhaps instead we have a policy that we do not
> change the behavior of existing tests in such significant ways, at
> least if they're tagged as being used by DefCore. I don't know --
> those are things we need to discuss.

 Sure I agree, this thread raises larger issues which need to be figured 
 out.
 But, that is probably an independent discussion.
>>>
>>> I’m beginning to wonder if we need to make DefCore use release
>>> branches then back-port bug-fixes and relevant features additions
>>> as necessary.

What I suggested when the TC decided on keeping the tests in tempest
was to keep them as a tempest plugin.

This would allow the tests to progress overtime, and be versioned,
so that def-core 201x.y would be equal to def-core-tempest-plugin
version 201x.y .

This allows the project developers to continue to develop tests that
match their vision, without causing unforeseen breakages in def-core.

It also allows the def-core tests to target a wider range - tests
that are run currently have no guarantee that they will run against
Kilo, but the def-core plugin could be tested against Kilo known good
clouds in its gate.

Is there any major blocker for moving them?

>> We should definitely have that conversation, to understand what
>> effect it would have both on Tempest and on DefCore.
>>
>
> While from a quick glance this would seem like it would solve some of the
> problems when you start to dig into it you'll see that it actually wouldn't,
> and would just end up causing more issues in the long run. Branchless tempest
> was originally started back at the icehouse release and was implemented to
> actually enforce the API is the same across release boundaries. We had hit 
> many
> issues where incompatibilities inadvertently were introduced into projects' 
> APIs
> which weren't caught because of divergence between tempest master and tempest
> stable/*. If you decide to branch you'll end up having to do the same thing or
> you'll risk the same types of regressions. Testing stable branches against
> master puts you in a weird place where upstream changes could break things for
> your branch and you'd never know until you tried to land something. From a
> tempest dev standpoint branching quite frankly doesn't make any sense.
>
> Also, the other thing to consider is that there wouldn't actually be anything 
> to
> branch on. Tempest always supports whatever releases the community is
> supporting. Every commit is tested against all the branches of OpenStack that
> still exist. When we EOL a stable branch there is no longer anything to run
> tests against. Assuming you're primarily motivated by the fact defcore 
> attempts
> to support branches that no longer have upstream support you wouldn't actually
> be able to do this by branching. When a branch is removed there isn't anything
> for you to test tempest changes against, and merging code without 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Morgan Fainberg]

On Wed, Jun 15, 2016 at 11:54 PM, Ken'ichi Ohmichi 
wrote:

> This discussion was expected when we implemented the Tempest patch,
> then I sent a mail to defcore comittee[1]
> As the above ml, "A DefCore Guideline typically covers three OpenStack
> releases".
> That means the latest guideline needs to cover Mitaka, Liberty and Kilo,
> right?
>
> In the Kilo development, we(nova team) have already considered
> additional properties are not good for the interoperability.
> And the stable_api.rst of [2] which is contained in Kilo says we need
> to implement new features without extensions.
> However, there are Kilo+ clouds which are extended with vendors' own
> extensions, right?
>
> My concern of allowing additional properties on interoperability tests is
> that
>  - users can move from pure OpenStack clouds to non-pure OpenStack
> clouds which implement vender specific properties
>  - but users cannot move from non-pure OpenStack clouds if users
> depend on the properties
> even if these clouds are certificated on the same interoperability tests.
>
>
The end goal is 100% to get everyone consistent with no "extra" data being
passed out of the APIs and certified on the same tests.

However, right now we have an issue where vendors/operators are lagging on
getting this cleaned up. Since this is the first round of certifications
(among other things), the proposal is to support/manage this in a way that
gives a bit more of a grace period while the deployers/operators finish
moving away from custom properties (as i understand it the ones affected
have communicated that they are working on meeting this goal; Chris, please
correct me if I am wrong).

Your concerns are spot on, and at the end of this "greylist" window ( at
the " 2017.01" defcore guideline ), this grace period will expire and
everyone will be expected to be compatible without the "Extra" data. Part
of the process of doing these programs is working to refine the process
(and sometimes make exceptions in the early stages) until the workflow
is established and understood. It is not expected to continue nor extend
the period beyond the firm end point Chris highlighted. I would not support
this proposal if it was open ended.

Cheers,
--Morgan


> Thanks
> Ken Ohmichi
>
> ---
> [1]:
> http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
> [2]: https://review.openstack.org/#/c/162912
>
> 2016-06-14 16:37 GMT-07:00 Chris Hoge :
> > Top posting one note and direct comments inline, I’m proposing
> > this as a member of the DefCore working group, but this
> > proposal itself has not been accepted as the forward course of
> > action by the working group. These are my own views as the
> > administrator of the program and not that of the working group
> > itself, which may independently reject the idea outside of the
> > response from the upstream devs.
> >
> > I posted a link to this thread to the DefCore mailing list to make
> > that working group aware of the outstanding issues.
> >
> > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
> wrote:
> >
> > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> >
> > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> >
> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> >
> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> >
> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> >
> > Last year, in response to Nova micro-versioning and extension updates[1],
> > the QA team added strict API schema checking to Tempest to ensure that
> > no additional properties were added to Nova API responses[2][3]. In the
> > last year, at least three vendors participating the the OpenStack Powered
> > Trademark program have been impacted by this change, two of which
> > reported this to the DefCore Working Group mailing list earlier this
> > year[4].
> >
> > The DefCore Working Group determines guidelines for the OpenStack Powered
> > program, which includes capabilities with associated functional tests
> > from Tempest that must be passed, and designated sections with associated
> > upstream code [5][6]. In determining these guidelines, the working group
> > attempts to balance the future direction of development with lagging
> > indicators of deployments and user adoption.
> >
> > After a tremendous amount of consideration, I believe that the DefCore
> > Working Group needs to implement a temporary waiver for the strict API
> > checking requirements that were introduced last year, to give downstream
> > deployers more time to catch up with the strict micro-versioning
> > requirements determined by the Nova/Compute team and enforced by the
> > Tempest/QA team.
> >
> >
> > I'm very much opposed to this being done. If we're actually concerned
> with
> > interoperability and verify that things behave in the same manner between
> > multiple
> > clouds then doing 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Ken'ichi Ohmichi]

This discussion was expected when we implemented the Tempest patch,
then I sent a mail to defcore comittee[1]
As the above ml, "A DefCore Guideline typically covers three OpenStack
releases".
That means the latest guideline needs to cover Mitaka, Liberty and Kilo, right?

In the Kilo development, we(nova team) have already considered
additional properties are not good for the interoperability.
And the stable_api.rst of [2] which is contained in Kilo says we need
to implement new features without extensions.
However, there are Kilo+ clouds which are extended with vendors' own
extensions, right?

My concern of allowing additional properties on interoperability tests is that
 - users can move from pure OpenStack clouds to non-pure OpenStack
clouds which implement vender specific properties
 - but users cannot move from non-pure OpenStack clouds if users
depend on the properties
even if these clouds are certificated on the same interoperability tests.

Thanks
Ken Ohmichi

---
[1]: 
http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
[2]: https://review.openstack.org/#/c/162912

2016-06-14 16:37 GMT-07:00 Chris Hoge :
> Top posting one note and direct comments inline, I’m proposing
> this as a member of the DefCore working group, but this
> proposal itself has not been accepted as the forward course of
> action by the working group. These are my own views as the
> administrator of the program and not that of the working group
> itself, which may independently reject the idea outside of the
> response from the upstream devs.
>
> I posted a link to this thread to the DefCore mailing list to make
> that working group aware of the outstanding issues.
>
> On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:
>
> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>
> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>
> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
>
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>
> Last year, in response to Nova micro-versioning and extension updates[1],
> the QA team added strict API schema checking to Tempest to ensure that
> no additional properties were added to Nova API responses[2][3]. In the
> last year, at least three vendors participating the the OpenStack Powered
> Trademark program have been impacted by this change, two of which
> reported this to the DefCore Working Group mailing list earlier this
> year[4].
>
> The DefCore Working Group determines guidelines for the OpenStack Powered
> program, which includes capabilities with associated functional tests
> from Tempest that must be passed, and designated sections with associated
> upstream code [5][6]. In determining these guidelines, the working group
> attempts to balance the future direction of development with lagging
> indicators of deployments and user adoption.
>
> After a tremendous amount of consideration, I believe that the DefCore
> Working Group needs to implement a temporary waiver for the strict API
> checking requirements that were introduced last year, to give downstream
> deployers more time to catch up with the strict micro-versioning
> requirements determined by the Nova/Compute team and enforced by the
> Tempest/QA team.
>
>
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between
> multiple
> clouds then doing this would be a big step backwards. The fundamental
> disconnect
> here is that the vendors who have implemented out of band extensions or were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**
>
>
> This is a temporary measure to address the fact that a large number
> of existing tests changed their behavior, rather than having new
> tests added to enforce this new requirement. The result is deployments
> that previously passed these tests may no longer pass, and in fact
> we have several cases where that's true with deployers who are
> trying to maintain their own standard of backwards-compatibility
> for their end users.
>
>
> That's not what happened though. The API hasn't changed and the tests
> haven't
> really changed either. We made our enforcement on Nova's APIs a bit stricter
> to
> ensure nothing unexpected appeared. For the most these tests work on any
> version
> of OpenStack. (we only test it in the gate on supported stable releases, but
> I
> don't expect things to have drastically shifted on older releases) It also
> doesn't matter which version of the API you run, v2.0 or v2.1. Literally,
> the
> only case it ever fails is when you run something extra, not from the
> community,
> either as an extension (which 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Morgan Fainberg]

On Jun 14, 2016 14:42, "Doug Hellmann"  wrote:
>
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > Last year, in response to Nova micro-versioning and extension
updates[1],
> > > > > the QA team added strict API schema checking to Tempest to ensure
that
> > > > > no additional properties were added to Nova API responses[2][3].
In the
> > > > > last year, at least three vendors participating the the OpenStack
Powered
> > > > > Trademark program have been impacted by this change, two of which
> > > > > reported this to the DefCore Working Group mailing list earlier
this year[4].
> > > > >
> > > > > The DefCore Working Group determines guidelines for the OpenStack
Powered
> > > > > program, which includes capabilities with associated functional
tests
> > > > > from Tempest that must be passed, and designated sections with
associated
> > > > > upstream code [5][6]. In determining these guidelines, the
working group
> > > > > attempts to balance the future direction of development with
lagging
> > > > > indicators of deployments and user adoption.
> > > > >
> > > > > After a tremendous amount of consideration, I believe that the
DefCore
> > > > > Working Group needs to implement a temporary waiver for the
strict API
> > > > > checking requirements that were introduced last year, to give
downstream
> > > > > deployers more time to catch up with the strict micro-versioning
> > > > > requirements determined by the Nova/Compute team and enforced by
the
> > > > > Tempest/QA team.
> > > >
> > > > I'm very much opposed to this being done. If we're actually
concerned with
> > > > interoperability and verify that things behave in the same manner
between multiple
> > > > clouds then doing this would be a big step backwards. The
fundamental disconnect
> > > > here is that the vendors who have implemented out of band
extensions or were
> > > > taking advantage of previously available places to inject extra
attributes
> > > > believe that doing so means they're interoperable, which is quite
far from
> > > > reality. **The API is not a place for vendor differentiation.**
> > >
> > > This is a temporary measure to address the fact that a large number
> > > of existing tests changed their behavior, rather than having new
> > > tests added to enforce this new requirement. The result is deployments
> > > that previously passed these tests may no longer pass, and in fact
> > > we have several cases where that's true with deployers who are
> > > trying to maintain their own standard of backwards-compatibility
> > > for their end users.
> >
> > That's not what happened though. The API hasn't changed and the tests
haven't
> > really changed either. We made our enforcement on Nova's APIs a bit
stricter to
> > ensure nothing unexpected appeared. For the most these tests work on
any version
> > of OpenStack. (we only test it in the gate on supported stable
releases, but I
> > don't expect things to have drastically shifted on older releases) It
also
> > doesn't matter which version of the API you run, v2.0 or v2.1.
Literally, the
> > only case it ever fails is when you run something extra, not from the
community,
> > either as an extension (which themselves are going away [1]) or another
service
> > that wraps nova or imitates nova. I'm personally not comfortable saying
those
> > extras are ever part of the OpenStack APIs.
> >
> > > We have basically three options.
> > >
> > > 1. Tell deployers who are trying to do the right for their immediate
> > >users that they can't use the trademark.
> > >
> > > 2. Flag the related tests or remove them from the DefCore enforcement
> > >suite entirely.
> > >
> > > 3. Be flexible about giving consumers of Tempest time to meet the
> > >new requirement by providing a way to disable the checks.
> > >
> > > Option 1 goes against our own backwards compatibility policies.
> >
> > I don't think backwards compatibility policies really apply to what
what define
> > as the set of tests that as a community we are saying a vendor has to
pass to
> > say they're OpenStack. From my perspective as a community we either
take a hard
> > stance on this and say to be considered an interoperable cloud (and to
get the
> > trademark) you have to actually have an interoperable product. We
slowly ratchet
> > up the requirements every 6 months, there isn't any implied backwards
> > compatibility in doing that. You passed in the past but not in the
newer stricter
> > guidelines.
> >
> > Also, even if I did think it applied, we're not talking about a change
which
> > would fall into breaking that. The change was introduced a year and
half ago
> > during kilo and landed a year ago during liberty:
> >
> > https://review.openstack.org/#/c/156130/
> >
> 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Daryl Walleck]

> -Original Message-
> From: GHANSHYAM MANN [mailto:ghanshyamm...@gmail.com]
> Sent: Wednesday, June 15, 2016 9:59 PM
> To: OpenStack Development Mailing List (not for usage questions)
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to
> disable some strict response checking for interop testing
> 
> On Wed, Jun 15, 2016 at 6:12 AM, Matthew Treinish <mtrein...@kortar.org>
> wrote:
> > On Tue, Jun 14, 2016 at 12:19:54PM -0700, Chris Hoge wrote:
> >>
> >> > On Jun 14, 2016, at 11:21 AM, Matthew Treinish <mtrein...@kortar.org>
> wrote:
> >> >
> >> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> >> >> Last year, in response to Nova micro-versioning and extension
> >> >> updates[1], the QA team added strict API schema checking to
> >> >> Tempest to ensure that no additional properties were added to Nova
> >> >> API responses[2][3]. In the last year, at least three vendors
> >> >> participating the the OpenStack Powered Trademark program have
> >> >> been impacted by this change, two of which reported this to the
> DefCore Working Group mailing list earlier this year[4].
> >> >>
> >> >> The DefCore Working Group determines guidelines for the OpenStack
> >> >> Powered program, which includes capabilities with associated
> >> >> functional tests from Tempest that must be passed, and designated
> >> >> sections with associated upstream code [5][6]. In determining
> >> >> these guidelines, the working group attempts to balance the future
> >> >> direction of development with lagging indicators of deployments and
> user adoption.
> >> >>
> >> >> After a tremendous amount of consideration, I believe that the
> >> >> DefCore Working Group needs to implement a temporary waiver for
> >> >> the strict API checking requirements that were introduced last
> >> >> year, to give downstream deployers more time to catch up with the
> >> >> strict micro-versioning requirements determined by the
> >> >> Nova/Compute team and enforced by the Tempest/QA team.
> >> >
> >> > I'm very much opposed to this being done. If we're actually
> >> > concerned with interoperability and verify that things behave in
> >> > the same manner between multiple clouds then doing this would be a
> >> > big step backwards. The fundamental disconnect here is that the
> >> > vendors who have implemented out of band extensions or were taking
> >> > advantage of previously available places to inject extra attributes
> >> > believe that doing so means they're interoperable, which is quite
> >> > far from reality. **The API is not a place for vendor
> >> > differentiation.**
> >>
> >> Yes, it’s bad practice, but it’s also a reality, and I honestly
> >> believe that vendors have received the message and are working on
> changing.
> >
> > They might be working on this, but this change was coming for quite
> > some time it shouldn't be a surprise to anyone at this point. I mean
> > seriously, it's been in tempest for 1 year, and it took 6months to
> > land. Also, lets say we set a hard deadline on this new option to disable 
> > the
> enforcement and enforce it.
> > Then we implement a similar change on keystone are we gonna have to do
> > the same thing again when vendors who have custom things running there
> fail.
> >
> >>
> >> > As a user of several clouds myself I can say that having random
> >> > gorp in a response makes it much more difficult to use my code
> >> > against multiple clouds. I have to determine which properties being
> >> > returned are specific to that vendor's cloud and if I actually need
> >> > to depend on them for anything it makes whatever code I'm writing
> >> > incompatible for using against any other cloud. (unless I special
> >> > case that block for each cloud) Sean Dague wrote a good post where a
> lot of this was covered a year ago when microversions was starting to pick up
> steam:
> >> >
> >> > https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2
> >> > <https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2>
> >> >
> >> > I'd recommend giving it a read, he explains the user first
> >> > perspective more clearly there.
> >> >
> &g

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[GHANSHYAM MANN]

On Wed, Jun 15, 2016 at 6:12 AM, Matthew Treinish  wrote:
> On Tue, Jun 14, 2016 at 12:19:54PM -0700, Chris Hoge wrote:
>>
>> > On Jun 14, 2016, at 11:21 AM, Matthew Treinish  
>> > wrote:
>> >
>> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> >> Last year, in response to Nova micro-versioning and extension updates[1],
>> >> the QA team added strict API schema checking to Tempest to ensure that
>> >> no additional properties were added to Nova API responses[2][3]. In the
>> >> last year, at least three vendors participating the the OpenStack Powered
>> >> Trademark program have been impacted by this change, two of which
>> >> reported this to the DefCore Working Group mailing list earlier this 
>> >> year[4].
>> >>
>> >> The DefCore Working Group determines guidelines for the OpenStack Powered
>> >> program, which includes capabilities with associated functional tests
>> >> from Tempest that must be passed, and designated sections with associated
>> >> upstream code [5][6]. In determining these guidelines, the working group
>> >> attempts to balance the future direction of development with lagging
>> >> indicators of deployments and user adoption.
>> >>
>> >> After a tremendous amount of consideration, I believe that the DefCore
>> >> Working Group needs to implement a temporary waiver for the strict API
>> >> checking requirements that were introduced last year, to give downstream
>> >> deployers more time to catch up with the strict micro-versioning
>> >> requirements determined by the Nova/Compute team and enforced by the
>> >> Tempest/QA team.
>> >
>> > I'm very much opposed to this being done. If we're actually concerned with
>> > interoperability and verify that things behave in the same manner between 
>> > multiple
>> > clouds then doing this would be a big step backwards. The fundamental 
>> > disconnect
>> > here is that the vendors who have implemented out of band extensions or 
>> > were
>> > taking advantage of previously available places to inject extra attributes
>> > believe that doing so means they're interoperable, which is quite far from
>> > reality. **The API is not a place for vendor differentiation.**
>>
>> Yes, it’s bad practice, but it’s also a reality, and I honestly believe that
>> vendors have received the message and are working on changing.
>
> They might be working on this, but this change was coming for quite some
> time it shouldn't be a surprise to anyone at this point. I mean seriously, 
> it's
> been in tempest for 1 year, and it took 6months to land. Also, lets say we set
> a hard deadline on this new option to disable the enforcement and enforce it.
> Then we implement a similar change on keystone are we gonna have to do the 
> same
> thing again when vendors who have custom things running there fail.
>
>>
>> > As a user of several clouds myself I can say that having random gorp in a
>> > response makes it much more difficult to use my code against multiple 
>> > clouds. I
>> > have to determine which properties being returned are specific to that 
>> > vendor's
>> > cloud and if I actually need to depend on them for anything it makes 
>> > whatever
>> > code I'm writing incompatible for using against any other cloud. (unless I
>> > special case that block for each cloud) Sean Dague wrote a good post where 
>> > a lot
>> > of this was covered a year ago when microversions was starting to pick up 
>> > steam:
>> >
>> > https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2 
>> > 
>> >
>> > I'd recommend giving it a read, he explains the user first perspective more
>> > clearly there.
>> >
>> > I believe Tempest in this case is doing the right thing from an 
>> > interoperability
>> > perspective and ensuring that the API is actually the API. Not an API with 
>> > extra
>> > bits a vendor decided to add.
>>
>> A few points on this, though. Right now, Nova is the only API that is
>> enforcing this, and the clients. While this may change in the
>> future, I don’t think it accurately represents the reality of what’s
>> happening in the ecosystem.
>
> This in itself doesn't make a difference. There is a disparity in the level of
> testing across all the projects. Nova happens to be further along in regards
> to api stability and testing things compared to a lot of projects, it's not
> really a surprise that they're the first for this to come up on. It's only a
> matter of time for other projects to follow nova's example and implement 
> similar
> enforcement.
>
>>
>> As mentioned before, we also need to balance the lagging nature of
>> DefCore as an interoperability guideline with the needs of testing
>> upstream changes. I’m not asking for a permanent change that
>> undermines the goals of Tempest for QA, rather a temporary
>> upstream modification that recognizes the challenges faced by
>> vendors in the market right now, and gives them 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > Top posting one note and direct comments inline, I’m proposing
> > this as a member of the DefCore working group, but this
> > proposal itself has not been accepted as the forward course of
> > action by the working group. These are my own views as the
> > administrator of the program and not that of the working group
> > itself, which may independently reject the idea outside of the
> > response from the upstream devs.
> > 
> > I posted a link to this thread to the DefCore mailing list to make
> > that working group aware of the outstanding issues.
> > 
> > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  
> > > wrote:
> > > 
> > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > >> Last year, in response to Nova micro-versioning and extension 
> > >> updates[1],
> > >> the QA team added strict API schema checking to Tempest to ensure 
> > >> that
> > >> no additional properties were added to Nova API responses[2][3]. In 
> > >> the
> > >> last year, at least three vendors participating the the OpenStack 
> > >> Powered
> > >> Trademark program have been impacted by this change, two of which
> > >> reported this to the DefCore Working Group mailing list earlier this 
> > >> year[4].
> > >> 
> > >> The DefCore Working Group determines guidelines for the OpenStack 
> > >> Powered
> > >> program, which includes capabilities with associated functional tests
> > >> from Tempest that must be passed, and designated sections with 
> > >> associated
> > >> upstream code [5][6]. In determining these guidelines, the working 
> > >> group
> > >> attempts to balance the future direction of development with lagging
> > >> indicators of deployments and user adoption.
> > >> 
> > >> After a tremendous amount of consideration, I believe that the 
> > >> DefCore
> > >> Working Group needs to implement a temporary waiver for the strict 
> > >> API
> > >> checking requirements that were introduced last year, to give 
> > >> downstream
> > >> deployers more time to catch up with the strict micro-versioning
> > >> requirements determined by the Nova/Compute team and enforced by the
> > >> Tempest/QA team.
> > > 
> > > I'm very much opposed to this being done. If we're actually concerned 
> > > with
> > > interoperability and verify that things behave in the same manner 
> > > between multiple
> > > clouds then doing this would be a big step backwards. The fundamental 
> > > disconnect
> > > here is that the vendors who have implemented out of band extensions 
> > > or were
> > > taking advantage of previously available places to inject extra 
> > > attributes
> > > believe that doing so means they're interoperable, which is quite far 
> > > from
> > > reality. **The API is not a place for vendor differentiation.**
> >  
> >  This is a temporary measure to address the fact that a large number
> >  of existing tests changed their behavior, rather than having new
> >  tests added to enforce this new requirement. The result is deployments
> >  that previously passed these tests may no longer pass, and in fact
> >  we have several cases where that's true with deployers who are
> >  trying to maintain their own standard of backwards-compatibility
> >  for their end users.
> > >>> 
> > >>> That's not what happened though. The API hasn't changed and the tests 
> > >>> haven't
> > >>> really changed either. We made our enforcement on Nova's APIs a bit 
> > >>> stricter to
> > >>> ensure nothing unexpected appeared. For the most these tests work on 
> > >>> any version
> > >>> of OpenStack. (we only test it in the gate on supported stable 
> > >>> releases, but I
> > >>> don't expect things to have drastically shifted on older releases) It 
> > >>> also
> > >>> doesn't matter which version of the API you run, v2.0 or v2.1. 
> > >>> Literally, the
> > >>> only case it ever fails is when you run something extra, not from the 
> > >>> community,
> > >>> either as an extension (which themselves are going away [1]) or another 
> > >>> service
> > >>> that wraps nova or imitates nova. I'm personally not comfortable saying 
> > >>> those
> > >>> extras are ever part of the OpenStack APIs.
> > >>> 
> >  We have basically three options.
> >  
> >  1. Tell deployers who are trying to do the right for their immediate
> >    users that they can't use the 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

> On Jun 15, 2016, at 8:01 AM, Sean Dague  wrote:
> 
> On 06/15/2016 12:14 AM, Mark Voelker wrote:
> 
>> 
>> It is perhaps important to note here that the DefCore seems to have two 
>> meanings to a lot of people I talk to today: it’s a mark of interoperability 
>> (the OpenStack Powered badge that says certain capabilities of this cloud 
>> behave like other clouds bearing the mark) and it gives a cloud the ability 
>> to call itself OpenStack (e.g. you can get a trademark/logo license 
>> agreement from the Foundation).  
>> 
>> The OpenStack Powered program currently covers Icehouse through Mitaka.  
>> Right now, that includes releases that were still on the Nova 2.0 API.  API 
>> extensions were a supported thing [1] back in 2.0 and it was even explicitly 
>> documented that they allowed for additional attributes in the responses and 
>> “vendor specific niche functionality [1]”.  The change to the Tempest tests 
>> [2] applied to the 2.0 API as well as 2.1 with the intent of preventing 
>> further changes from getting into the 2.0 API at the gate, which totally 
>> makes sense as a gate test.  If those same tests are used for DefCore 
>> purposes, it does change what vendors need to do to be compliant with the 
>> Guidelines rather immediately--even on older releases of OpenStack using 
>> 2.0, which could be problematic (as noted elsewhere already [3]).
> 
> Right, that's fair. And part of why I think the pass* makes sense.
> Liberty is the introduction of microversions on by default for clouds
> from Nova upstream configs.
> 
>> So, through the interoperability lens: I think many folks acknowledge that 
>> supporting extensions lead to a lot of variance between clouds, and that was 
>> Not So Awesome for interoperability.  IIRC part of the rationale for 
>> switching to microversions with a single monotonic counter and deprecating 
>> extensions [4] was to set a course for eliminating a lot of that behavioral 
>> variance.
>> 
>> From the “ability to call yourself OpenStack” lens: it feels sort of wrong 
>> to tell a cloud that it can’t claim to be OpenStack because it’s running a 
>> version that falls within the bounds of the Powered program with the 2.0 API 
>> (when extensions weren't deprecated) and using the extension mechanism that 
>> 2.0 supported for years.
> 
> To be clear, extensions weren't a part of the 2.0 API, they were a part
> of the infrastructure. It's a subtle but different point. Nova still
> supports the 2.0 API, but on different infrastructure, which
> doesn't/won't support extensions.
> 
> Are people registering new Kilo (or earlier) clouds in the system today?
> By the time folks get to Newton, none of that is going to work anyway in
> code.

Yes.  As recently as January we had a Juno-based public cloud run into this 
issue [1].  And, just to highlight again: there's a pretty long tail here since 
the OpenStack Powered program covers a lot of releases.  

[1] 
http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html

> 
> In an ideal world product teams would be close enough to upstream code
> and changes to see all this coming and be on top of it. In the real
> world, a lot of these teams are like a year (or more) behind, which

+1.  If folks aren’t aware, it may be worth pointing out that as of the User 
Survey from two months ago, Kilo was the most popular release and Juno was the 
third most popular [2].  I don’t presume to know why so many deployments are on 
older releases, but I suspect the rationale is different for different folks.  
Maybe for some it’s because people are choosing to used products and a lot of 
those are based on older releases.  Maybe for others it’s because people are 
deliberately choosing older stable branches.  Maybe for some it’s because they 
they’ve decided that upgrading every six months or following master more 
closely just isn’t a good strategy for them.  Maybe for others it’s something 
else entirely.  But again, you’re right: in the real world the tail is long 
today.

[2] https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf 
(figure 4.3/4.4)

> actually makes Defcore with a pass* an ideal alternative communication
> channel to express that products are coming up to a cliff, and should
> start working on plans now.

Agreed.

> 
>> I think that’s part of what makes this issue tricky for a lot of folks.
>> 
>> [1] http://docs.openstack.org/developer/nova/v2/extensions.html
> 
> It's an unfortunate accident of bugs in our publishing system that that
> URL still exists. That was deleted in Oct 2015. I'll look at getting it
> properly cleaned up.
> 

Awesome, thanks Sean!  I’ll make a point to comb around and see if there are 
other such references that we should clean up too. 

Just to be clear though, my main point isn’t that the doc still exists today, 
it’s that it existed.  In the relatively recent past (from a long-tail 
perspective) extensions were documented as an 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> Top posting one note and direct comments inline, I’m proposing
> this as a member of the DefCore working group, but this
> proposal itself has not been accepted as the forward course of
> action by the working group. These are my own views as the
> administrator of the program and not that of the working group
> itself, which may independently reject the idea outside of the
> response from the upstream devs.
> 
> I posted a link to this thread to the DefCore mailing list to make
> that working group aware of the outstanding issues.
> 
> > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:
> > 
> > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> >> Last year, in response to Nova micro-versioning and extension 
> >> updates[1],
> >> the QA team added strict API schema checking to Tempest to ensure that
> >> no additional properties were added to Nova API responses[2][3]. In the
> >> last year, at least three vendors participating the the OpenStack 
> >> Powered
> >> Trademark program have been impacted by this change, two of which
> >> reported this to the DefCore Working Group mailing list earlier this 
> >> year[4].
> >> 
> >> The DefCore Working Group determines guidelines for the OpenStack 
> >> Powered
> >> program, which includes capabilities with associated functional tests
> >> from Tempest that must be passed, and designated sections with 
> >> associated
> >> upstream code [5][6]. In determining these guidelines, the working 
> >> group
> >> attempts to balance the future direction of development with lagging
> >> indicators of deployments and user adoption.
> >> 
> >> After a tremendous amount of consideration, I believe that the DefCore
> >> Working Group needs to implement a temporary waiver for the strict API
> >> checking requirements that were introduced last year, to give 
> >> downstream
> >> deployers more time to catch up with the strict micro-versioning
> >> requirements determined by the Nova/Compute team and enforced by the
> >> Tempest/QA team.
> > 
> > I'm very much opposed to this being done. If we're actually concerned 
> > with
> > interoperability and verify that things behave in the same manner 
> > between multiple
> > clouds then doing this would be a big step backwards. The fundamental 
> > disconnect
> > here is that the vendors who have implemented out of band extensions or 
> > were
> > taking advantage of previously available places to inject extra 
> > attributes
> > believe that doing so means they're interoperable, which is quite far 
> > from
> > reality. **The API is not a place for vendor differentiation.**
>  
>  This is a temporary measure to address the fact that a large number
>  of existing tests changed their behavior, rather than having new
>  tests added to enforce this new requirement. The result is deployments
>  that previously passed these tests may no longer pass, and in fact
>  we have several cases where that's true with deployers who are
>  trying to maintain their own standard of backwards-compatibility
>  for their end users.
> >>> 
> >>> That's not what happened though. The API hasn't changed and the tests 
> >>> haven't
> >>> really changed either. We made our enforcement on Nova's APIs a bit 
> >>> stricter to
> >>> ensure nothing unexpected appeared. For the most these tests work on any 
> >>> version
> >>> of OpenStack. (we only test it in the gate on supported stable releases, 
> >>> but I
> >>> don't expect things to have drastically shifted on older releases) It also
> >>> doesn't matter which version of the API you run, v2.0 or v2.1. Literally, 
> >>> the
> >>> only case it ever fails is when you run something extra, not from the 
> >>> community,
> >>> either as an extension (which themselves are going away [1]) or another 
> >>> service
> >>> that wraps nova or imitates nova. I'm personally not comfortable saying 
> >>> those
> >>> extras are ever part of the OpenStack APIs.
> >>> 
>  We have basically three options.
>  
>  1. Tell deployers who are trying to do the right for their immediate
>    users that they can't use the trademark.
>  
>  2. Flag the related tests or remove them from the DefCore enforcement
>    suite entirely.
>  
>  3. Be flexible about giving consumers of Tempest time to meet the
>    new requirement by providing a way to disable the checks.
>  
>  Option 1 goes against our own 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/15/2016 12:14 AM, Mark Voelker wrote:

> 
> It is perhaps important to note here that the DefCore seems to have two 
> meanings to a lot of people I talk to today: it’s a mark of interoperability 
> (the OpenStack Powered badge that says certain capabilities of this cloud 
> behave like other clouds bearing the mark) and it gives a cloud the ability 
> to call itself OpenStack (e.g. you can get a trademark/logo license agreement 
> from the Foundation).  
> 
> The OpenStack Powered program currently covers Icehouse through Mitaka.  
> Right now, that includes releases that were still on the Nova 2.0 API.  API 
> extensions were a supported thing [1] back in 2.0 and it was even explicitly 
> documented that they allowed for additional attributes in the responses and 
> “vendor specific niche functionality [1]”.  The change to the Tempest tests 
> [2] applied to the 2.0 API as well as 2.1 with the intent of preventing 
> further changes from getting into the 2.0 API at the gate, which totally 
> makes sense as a gate test.  If those same tests are used for DefCore 
> purposes, it does change what vendors need to do to be compliant with the 
> Guidelines rather immediately--even on older releases of OpenStack using 2.0, 
> which could be problematic (as noted elsewhere already [3]).

Right, that's fair. And part of why I think the pass* makes sense.
Liberty is the introduction of microversions on by default for clouds
from Nova upstream configs.

> So, through the interoperability lens: I think many folks acknowledge that 
> supporting extensions lead to a lot of variance between clouds, and that was 
> Not So Awesome for interoperability.  IIRC part of the rationale for 
> switching to microversions with a single monotonic counter and deprecating 
> extensions [4] was to set a course for eliminating a lot of that behavioral 
> variance.
> 
> From the “ability to call yourself OpenStack” lens: it feels sort of wrong to 
> tell a cloud that it can’t claim to be OpenStack because it’s running a 
> version that falls within the bounds of the Powered program with the 2.0 API 
> (when extensions weren't deprecated) and using the extension mechanism that 
> 2.0 supported for years.

To be clear, extensions weren't a part of the 2.0 API, they were a part
of the infrastructure. It's a subtle but different point. Nova still
supports the 2.0 API, but on different infrastructure, which
doesn't/won't support extensions.

Are people registering new Kilo (or earlier) clouds in the system today?
By the time folks get to Newton, none of that is going to work anyway in
code.

In an ideal world product teams would be close enough to upstream code
and changes to see all this coming and be on top of it. In the real
world, a lot of these teams are like a year (or more) behind, which
actually makes Defcore with a pass* an ideal alternative communication
channel to express that products are coming up to a cliff, and should
start working on plans now.

> I think that’s part of what makes this issue tricky for a lot of folks.
> 
> [1] http://docs.openstack.org/developer/nova/v2/extensions.html

It's an unfortunate accident of bugs in our publishing system that that
URL still exists. That was deleted in Oct 2015. I'll look at getting it
properly cleaned up.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Thierry Carrez]

Sean Dague wrote:

On 06/14/2016 07:28 PM, Monty Taylor wrote:

On 06/14/2016 05:42 PM, Doug Hellmann wrote:



I think this is the most important thing to me as it relates to this.
I'm obviously a huge proponent of clouds behaving more samely. But I
also think that, as Doug nicely describes above, we've sort of backed in
to removing something without a deprecation window ... largely because
of the complexities involved with the system here - and I'd like to make
sure that when we are being clear about behavior changes that we give
the warning period so that people can adapt.


I also think that "pass" to "pass *"  is useful social incentive. While
I think communication of this new direction has happened pretty broadly,
organizations are complex places, and it didn't filter everywhere it
needed to with the urgency that was probably needed.

"pass *"  * - with a greylist which goes away in 6 months

Will hopefully be a reasonable enough push to get the behavior we want,
which is everyone publishing the same interface.


I like Chris's suggestion of requiring vendors to submit the grey-list 
of APIs with additional response data, and publish that *very visibly* 
on their marketplace entry. That will immediately make it very clear how 
they are different, provide a strong social incentive to fix it in a 
reasonable timeframe, while stating as a community that such deviation 
is not acceptable long-term.


We agree on the end goal: we just have to choose between "pass fail 
pass" and "pass pass* pass" as a way to achieve it. I feel like the 
latter is a shorter path to the desired goal, and reduces the 
possibility of a "pass fail f*k-this-shit" rage outcome.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

On Jun 14, 2016, at 7:28 PM, Monty Taylor  wrote:
> 
> On 06/14/2016 05:42 PM, Doug Hellmann wrote:
>> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
 Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> Last year, in response to Nova micro-versioning and extension updates[1],
>> the QA team added strict API schema checking to Tempest to ensure that
>> no additional properties were added to Nova API responses[2][3]. In the
>> last year, at least three vendors participating the the OpenStack Powered
>> Trademark program have been impacted by this change, two of which
>> reported this to the DefCore Working Group mailing list earlier this 
>> year[4].
>> 
>> The DefCore Working Group determines guidelines for the OpenStack Powered
>> program, which includes capabilities with associated functional tests
>> from Tempest that must be passed, and designated sections with associated
>> upstream code [5][6]. In determining these guidelines, the working group
>> attempts to balance the future direction of development with lagging
>> indicators of deployments and user adoption.
>> 
>> After a tremendous amount of consideration, I believe that the DefCore
>> Working Group needs to implement a temporary waiver for the strict API
>> checking requirements that were introduced last year, to give downstream
>> deployers more time to catch up with the strict micro-versioning
>> requirements determined by the Nova/Compute team and enforced by the
>> Tempest/QA team.
> 
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between 
> multiple
> clouds then doing this would be a big step backwards. The fundamental 
> disconnect
> here is that the vendors who have implemented out of band extensions or 
> were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**
 
 This is a temporary measure to address the fact that a large number
 of existing tests changed their behavior, rather than having new
 tests added to enforce this new requirement. The result is deployments
 that previously passed these tests may no longer pass, and in fact
 we have several cases where that's true with deployers who are
 trying to maintain their own standard of backwards-compatibility
 for their end users.
>>> 
>>> That's not what happened though. The API hasn't changed and the tests 
>>> haven't
>>> really changed either. We made our enforcement on Nova's APIs a bit 
>>> stricter to
>>> ensure nothing unexpected appeared. For the most these tests work on any 
>>> version
>>> of OpenStack. (we only test it in the gate on supported stable releases, 
>>> but I
>>> don't expect things to have drastically shifted on older releases) It also
>>> doesn't matter which version of the API you run, v2.0 or v2.1. Literally, 
>>> the
>>> only case it ever fails is when you run something extra, not from the 
>>> community,
>>> either as an extension (which themselves are going away [1]) or another 
>>> service
>>> that wraps nova or imitates nova. I'm personally not comfortable saying 
>>> those
>>> extras are ever part of the OpenStack APIs.
>>> 
 We have basically three options.
 
 1. Tell deployers who are trying to do the right for their immediate
   users that they can't use the trademark.
 
 2. Flag the related tests or remove them from the DefCore enforcement
   suite entirely.
 
 3. Be flexible about giving consumers of Tempest time to meet the
   new requirement by providing a way to disable the checks.
 
 Option 1 goes against our own backwards compatibility policies.
>>> 
>>> I don't think backwards compatibility policies really apply to what what 
>>> define
>>> as the set of tests that as a community we are saying a vendor has to pass 
>>> to
>>> say they're OpenStack. From my perspective as a community we either take a 
>>> hard
>>> stance on this and say to be considered an interoperable cloud (and to get 
>>> the
>>> trademark) you have to actually have an interoperable product. We slowly 
>>> ratchet
>>> up the requirements every 6 months, there isn't any implied backwards
>>> compatibility in doing that. You passed in the past but not in the newer 
>>> stricter
>>> guidelines.
>>> 
>>> Also, even if I did think it applied, we're not talking about a change which
>>> would fall into breaking that. The change was introduced a year and half ago
>>> during kilo and landed a year ago during 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/14/2016 07:28 PM, Monty Taylor wrote:

On 06/14/2016 05:42 PM, Doug Hellmann wrote:



I think this is the most important thing to me as it relates to this.
I'm obviously a huge proponent of clouds behaving more samely. But I
also think that, as Doug nicely describes above, we've sort of backed in
to removing something without a deprecation window ... largely because
of the complexities involved with the system here - and I'd like to make
sure that when we are being clear about behavior changes that we give
the warning period so that people can adapt.


I also think that "pass" to "pass *"  is useful social incentive. While 
I think communication of this new direction has happened pretty broadly, 
organizations are complex places, and it didn't filter everywhere it 
needed to with the urgency that was probably needed.


"pass *"  * - with a greylist which goes away in 6 months

Will hopefully be a reasonable enough push to get the behavior we want, 
which is everyone publishing the same interface.


-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Chris Hoge's message of 2016-06-14 16:19:11 -0700:
> 
> > On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
> > 
> > On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
> > 
> >> But, if we add another possible state on the defcore side like conditional 
> >> pass,
> >> warning, yellow, etc. (the name doesn't matter) which is used to indicate 
> >> that
> >> things on product X could only pass when strict validation was disabled 
> >> (and
> >> be clear about where and why) then my concerns would be alleviated. I just 
> >> do
> >> not want this to end up not being visible to end users trying to evaluate
> >> interoperability of different clouds using the test results.
> > 
> > +1
> > 
> > Don't fail them, but don't cover up their incompatibility, either.
> > -- Ed Leafe
> 
> That’s not my proposal. My requirement is that vendors who want to do this
> state exactly which APIs are sending back additional data, and that this
> information be published.

I am reading the responses here as very much supporting that.

Doug

> 
> There are different levels of incompatibility. A response with additional data
> that can be safely ignored is different from a changed response that would
> cause a client to fail.
> 
> One would hope that micro-versions would be able to address this exact
> issue for vendors by giving them a means to propose optional but 
> well-defined API response additions (not extensions) that are defined
> upstream and usable by all vendors. If it’s not too off topic, can someone
> from the Nova team explain how something like that would work (if it
> would at all)?
> 
> -Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

Top posting one note and direct comments inline, I’m proposing
this as a member of the DefCore working group, but this
proposal itself has not been accepted as the forward course of
action by the working group. These are my own views as the
administrator of the program and not that of the working group
itself, which may independently reject the idea outside of the
response from the upstream devs.

I posted a link to this thread to the DefCore mailing list to make
that working group aware of the outstanding issues.

> On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:
> 
> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
 Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> Last year, in response to Nova micro-versioning and extension updates[1],
>> the QA team added strict API schema checking to Tempest to ensure that
>> no additional properties were added to Nova API responses[2][3]. In the
>> last year, at least three vendors participating the the OpenStack Powered
>> Trademark program have been impacted by this change, two of which
>> reported this to the DefCore Working Group mailing list earlier this 
>> year[4].
>> 
>> The DefCore Working Group determines guidelines for the OpenStack Powered
>> program, which includes capabilities with associated functional tests
>> from Tempest that must be passed, and designated sections with associated
>> upstream code [5][6]. In determining these guidelines, the working group
>> attempts to balance the future direction of development with lagging
>> indicators of deployments and user adoption.
>> 
>> After a tremendous amount of consideration, I believe that the DefCore
>> Working Group needs to implement a temporary waiver for the strict API
>> checking requirements that were introduced last year, to give downstream
>> deployers more time to catch up with the strict micro-versioning
>> requirements determined by the Nova/Compute team and enforced by the
>> Tempest/QA team.
> 
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between 
> multiple
> clouds then doing this would be a big step backwards. The fundamental 
> disconnect
> here is that the vendors who have implemented out of band extensions or 
> were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**
 
 This is a temporary measure to address the fact that a large number
 of existing tests changed their behavior, rather than having new
 tests added to enforce this new requirement. The result is deployments
 that previously passed these tests may no longer pass, and in fact
 we have several cases where that's true with deployers who are
 trying to maintain their own standard of backwards-compatibility
 for their end users.
>>> 
>>> That's not what happened though. The API hasn't changed and the tests 
>>> haven't
>>> really changed either. We made our enforcement on Nova's APIs a bit 
>>> stricter to
>>> ensure nothing unexpected appeared. For the most these tests work on any 
>>> version
>>> of OpenStack. (we only test it in the gate on supported stable releases, 
>>> but I
>>> don't expect things to have drastically shifted on older releases) It also
>>> doesn't matter which version of the API you run, v2.0 or v2.1. Literally, 
>>> the
>>> only case it ever fails is when you run something extra, not from the 
>>> community,
>>> either as an extension (which themselves are going away [1]) or another 
>>> service
>>> that wraps nova or imitates nova. I'm personally not comfortable saying 
>>> those
>>> extras are ever part of the OpenStack APIs.
>>> 
 We have basically three options.
 
 1. Tell deployers who are trying to do the right for their immediate
   users that they can't use the trademark.
 
 2. Flag the related tests or remove them from the DefCore enforcement
   suite entirely.
 
 3. Be flexible about giving consumers of Tempest time to meet the
   new requirement by providing a way to disable the checks.
 
 Option 1 goes against our own backwards compatibility policies.
>>> 
>>> I don't think backwards compatibility policies really apply to what what 
>>> define
>>> as the set of tests that as a community we are saying a vendor has to pass 
>>> to
>>> say they're OpenStack. From my perspective as a community we either take a 
>>> hard
>>> stance on this and say to 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Monty Taylor]

On 06/14/2016 05:42 PM, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
 On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> Last year, in response to Nova micro-versioning and extension updates[1],
> the QA team added strict API schema checking to Tempest to ensure that
> no additional properties were added to Nova API responses[2][3]. In the
> last year, at least three vendors participating the the OpenStack Powered
> Trademark program have been impacted by this change, two of which
> reported this to the DefCore Working Group mailing list earlier this 
> year[4].
>
> The DefCore Working Group determines guidelines for the OpenStack Powered
> program, which includes capabilities with associated functional tests
> from Tempest that must be passed, and designated sections with associated
> upstream code [5][6]. In determining these guidelines, the working group
> attempts to balance the future direction of development with lagging
> indicators of deployments and user adoption.
>
> After a tremendous amount of consideration, I believe that the DefCore
> Working Group needs to implement a temporary waiver for the strict API
> checking requirements that were introduced last year, to give downstream
> deployers more time to catch up with the strict micro-versioning
> requirements determined by the Nova/Compute team and enforced by the
> Tempest/QA team.

 I'm very much opposed to this being done. If we're actually concerned with
 interoperability and verify that things behave in the same manner between 
 multiple
 clouds then doing this would be a big step backwards. The fundamental 
 disconnect
 here is that the vendors who have implemented out of band extensions or 
 were
 taking advantage of previously available places to inject extra attributes
 believe that doing so means they're interoperable, which is quite far from
 reality. **The API is not a place for vendor differentiation.**
>>>
>>> This is a temporary measure to address the fact that a large number
>>> of existing tests changed their behavior, rather than having new
>>> tests added to enforce this new requirement. The result is deployments
>>> that previously passed these tests may no longer pass, and in fact
>>> we have several cases where that's true with deployers who are
>>> trying to maintain their own standard of backwards-compatibility
>>> for their end users.
>>
>> That's not what happened though. The API hasn't changed and the tests haven't
>> really changed either. We made our enforcement on Nova's APIs a bit stricter 
>> to
>> ensure nothing unexpected appeared. For the most these tests work on any 
>> version
>> of OpenStack. (we only test it in the gate on supported stable releases, but 
>> I
>> don't expect things to have drastically shifted on older releases) It also
>> doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the
>> only case it ever fails is when you run something extra, not from the 
>> community,
>> either as an extension (which themselves are going away [1]) or another 
>> service
>> that wraps nova or imitates nova. I'm personally not comfortable saying those
>> extras are ever part of the OpenStack APIs.
>>
>>> We have basically three options.
>>>
>>> 1. Tell deployers who are trying to do the right for their immediate
>>>users that they can't use the trademark.
>>>
>>> 2. Flag the related tests or remove them from the DefCore enforcement
>>>suite entirely.
>>>
>>> 3. Be flexible about giving consumers of Tempest time to meet the
>>>new requirement by providing a way to disable the checks.
>>>
>>> Option 1 goes against our own backwards compatibility policies.
>>
>> I don't think backwards compatibility policies really apply to what what 
>> define
>> as the set of tests that as a community we are saying a vendor has to pass to
>> say they're OpenStack. From my perspective as a community we either take a 
>> hard
>> stance on this and say to be considered an interoperable cloud (and to get 
>> the
>> trademark) you have to actually have an interoperable product. We slowly 
>> ratchet
>> up the requirements every 6 months, there isn't any implied backwards
>> compatibility in doing that. You passed in the past but not in the newer 
>> stricter
>> guidelines.
>>
>> Also, even if I did think it applied, we're not talking about a change which
>> would fall into breaking that. The change was introduced a year and half ago
>> during kilo and landed a year ago during liberty:
>>
>> https://review.openstack.org/#/c/156130/
>>
>> That's way longer than our normal deprecation period of 3 months and a 
>> release
>> boundary.
>>
>>>
>>> Option 2 gives us no winners and 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
> 
> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
> 
>> But, if we add another possible state on the defcore side like conditional 
>> pass,
>> warning, yellow, etc. (the name doesn't matter) which is used to indicate 
>> that
>> things on product X could only pass when strict validation was disabled (and
>> be clear about where and why) then my concerns would be alleviated. I just do
>> not want this to end up not being visible to end users trying to evaluate
>> interoperability of different clouds using the test results.
> 
> +1
> 
> Don't fail them, but don't cover up their incompatibility, either.
> -- Ed Leafe

That’s not my proposal. My requirement is that vendors who want to do this
state exactly which APIs are sending back additional data, and that this
information be published.

There are different levels of incompatibility. A response with additional data
that can be safely ignored is different from a changed response that would
cause a client to fail.

One would hope that micro-versions would be able to address this exact
issue for vendors by giving them a means to propose optional but 
well-defined API response additions (not extensions) that are defined
upstream and usable by all vendors. If it’s not too off topic, can someone
from the Nova team explain how something like that would work (if it
would at all)?

-Chris
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Edward Leafe]

On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:

> But, if we add another possible state on the defcore side like conditional 
> pass,
> warning, yellow, etc. (the name doesn't matter) which is used to indicate that
> things on product X could only pass when strict validation was disabled (and
> be clear about where and why) then my concerns would be alleviated. I just do
> not want this to end up not being visible to end users trying to evaluate
> interoperability of different clouds using the test results.

+1

Don't fail them, but don't cover up their incompatibility, either.

-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > Last year, in response to Nova micro-versioning and extension 
> > > > > updates[1],
> > > > > the QA team added strict API schema checking to Tempest to ensure that
> > > > > no additional properties were added to Nova API responses[2][3]. In 
> > > > > the
> > > > > last year, at least three vendors participating the the OpenStack 
> > > > > Powered
> > > > > Trademark program have been impacted by this change, two of which
> > > > > reported this to the DefCore Working Group mailing list earlier this 
> > > > > year[4].
> > > > > 
> > > > > The DefCore Working Group determines guidelines for the OpenStack 
> > > > > Powered
> > > > > program, which includes capabilities with associated functional tests
> > > > > from Tempest that must be passed, and designated sections with 
> > > > > associated
> > > > > upstream code [5][6]. In determining these guidelines, the working 
> > > > > group
> > > > > attempts to balance the future direction of development with lagging
> > > > > indicators of deployments and user adoption.
> > > > > 
> > > > > After a tremendous amount of consideration, I believe that the DefCore
> > > > > Working Group needs to implement a temporary waiver for the strict API
> > > > > checking requirements that were introduced last year, to give 
> > > > > downstream
> > > > > deployers more time to catch up with the strict micro-versioning
> > > > > requirements determined by the Nova/Compute team and enforced by the
> > > > > Tempest/QA team.
> > > > 
> > > > I'm very much opposed to this being done. If we're actually concerned 
> > > > with
> > > > interoperability and verify that things behave in the same manner 
> > > > between multiple
> > > > clouds then doing this would be a big step backwards. The fundamental 
> > > > disconnect
> > > > here is that the vendors who have implemented out of band extensions or 
> > > > were
> > > > taking advantage of previously available places to inject extra 
> > > > attributes
> > > > believe that doing so means they're interoperable, which is quite far 
> > > > from
> > > > reality. **The API is not a place for vendor differentiation.**
> > > 
> > > This is a temporary measure to address the fact that a large number
> > > of existing tests changed their behavior, rather than having new
> > > tests added to enforce this new requirement. The result is deployments
> > > that previously passed these tests may no longer pass, and in fact
> > > we have several cases where that's true with deployers who are
> > > trying to maintain their own standard of backwards-compatibility
> > > for their end users.
> > 
> > That's not what happened though. The API hasn't changed and the tests 
> > haven't
> > really changed either. We made our enforcement on Nova's APIs a bit 
> > stricter to
> > ensure nothing unexpected appeared. For the most these tests work on any 
> > version
> > of OpenStack. (we only test it in the gate on supported stable releases, 
> > but I
> > don't expect things to have drastically shifted on older releases) It also
> > doesn't matter which version of the API you run, v2.0 or v2.1. Literally, 
> > the
> > only case it ever fails is when you run something extra, not from the 
> > community,
> > either as an extension (which themselves are going away [1]) or another 
> > service
> > that wraps nova or imitates nova. I'm personally not comfortable saying 
> > those
> > extras are ever part of the OpenStack APIs.
> >
> > > We have basically three options.
> > > 
> > > 1. Tell deployers who are trying to do the right for their immediate
> > >users that they can't use the trademark.
> > > 
> > > 2. Flag the related tests or remove them from the DefCore enforcement
> > >suite entirely.
> > > 
> > > 3. Be flexible about giving consumers of Tempest time to meet the
> > >new requirement by providing a way to disable the checks.
> > > 
> > > Option 1 goes against our own backwards compatibility policies.
> > 
> > I don't think backwards compatibility policies really apply to what what 
> > define
> > as the set of tests that as a community we are saying a vendor has to pass 
> > to
> > say they're OpenStack. From my perspective as a community we either take a 
> > hard
> > stance on this and say to be considered an interoperable cloud (and to get 
> > the
> > trademark) you have to actually have an interoperable product. We slowly 
> > ratchet
> > up the requirements every 6 months, there isn't any implied backwards
> > compatibility in doing that. You passed in the past but not in the newer 
> > stricter
> > guidelines.
> > 
> > Also, even if I did think it applied, we're not 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Sean Dague's message of 2016-06-14 17:29:06 -0400:
> On 06/14/2016 01:57 PM, Chris Hoge wrote:
> 
> > 
> > My proposal for addressing this problem approaches it at two levels:
> > 
> > * For the short term, I will submit a blueprint and patch to tempest that
> >   allows configuration of a grey-list of Nova APIs where strict response
> >   checking on additional properties will be disabled. So, for example,
> >   if the 'create  servers' API call returned extra properties on that call,
> >   the strict checking on this line[8] would be disabled at runtime.
> >   Use of this code path will emit a deprecation warning, and the
> >   code will be scheduled for removal in 2017 directly after the release
> >   of the 2017.01 guideline. Vendors would be required so submit the
> >   grey-list of APIs with additional response data that would be
> >   published to their marketplace entry.
> 
> To understand more. Will there be a visible asterisk with their
> registration that says they require a grey-list?
> 
> > * Longer term, vendors will be expected to work with upstream to update
> >   the API for returning additional data that is compatible with
> >   API micro-versioning as defined by the Nova team, and the waiver would
> >   no longer be allowed after the release of the 2017.01 guideline.
> > 
> > For the next half-year, I feel that this approach strengthens 
> > interoperability
> > by accurately capturing the current state of OpenStack deployments and
> > client tools. Before this change, additional properties on responses
> > weren't explicitly disallowed, and vendors and deployers took advantage
> > of this in production. While this is behavior that the Nova and QA teams
> > want to stop, it will take a bit more time to reach downstream. Also, as
> > of right now, as far as I know the only client that does strict response
> > checking for Nova responses is the Tempest client. Currently, additional
> > properties in responses are ignored and do not break existing client
> > functionality. There is currently little to no harm done to downstream
> > users by temporarily allowing additional data to be returned in responses.
> 
> In general I'm ok with this, as long as three things are true:
> 
> 1) registrations that need the grey list are visually indicated quite
> clearly and publicly that they needed it to pass.

I like that. Chris' proposal was that the information would need to be
submitted with the application, and I think publishing it makes sense.
I'd like to see the whole list, either which APIs had to be flagged or
at least which tests, whichever we can do.

> 
> 2) 2017.01 is a firm cutoff.
> 
> 3) We have evidence that folks that are having challenges with the
> strict enforcement have made getting compliant a top priority.
> 
> 
> 3 is the one where I don't have any data either way. But I didn't see
> any specs submissions (which are required for API changes in Nova) for
> Newton that would indicate anyone is working on this. For 2017 to be a
> hard stop, that means folks are either deleting this from their
> interface, or proposing in Ocata. Which is a really short runway if this
> stuff isn't super straight forward and already upstream agreed.
> 
> So I'm provisionally ok with this, if folks in the know feel like 3 is
> covered.
> 
> -Sean
> 
> P.S. The Tempest changes pretty much just anticipate the Nova changes
> which are deleting all these facilities in Newton -
> https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/api-no-more-extensions.html
> - so in some ways we aren't doing folks a ton of favors letting them
> delay too far because they are about to hit a brick wall on the code side.
> 
> -Sean
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > Last year, in response to Nova micro-versioning and extension 
> > > > updates[1],
> > > > the QA team added strict API schema checking to Tempest to ensure that
> > > > no additional properties were added to Nova API responses[2][3]. In the
> > > > last year, at least three vendors participating the the OpenStack 
> > > > Powered
> > > > Trademark program have been impacted by this change, two of which
> > > > reported this to the DefCore Working Group mailing list earlier this 
> > > > year[4].
> > > > 
> > > > The DefCore Working Group determines guidelines for the OpenStack 
> > > > Powered
> > > > program, which includes capabilities with associated functional tests
> > > > from Tempest that must be passed, and designated sections with 
> > > > associated
> > > > upstream code [5][6]. In determining these guidelines, the working group
> > > > attempts to balance the future direction of development with lagging
> > > > indicators of deployments and user adoption.
> > > > 
> > > > After a tremendous amount of consideration, I believe that the DefCore
> > > > Working Group needs to implement a temporary waiver for the strict API
> > > > checking requirements that were introduced last year, to give downstream
> > > > deployers more time to catch up with the strict micro-versioning
> > > > requirements determined by the Nova/Compute team and enforced by the
> > > > Tempest/QA team.
> > > 
> > > I'm very much opposed to this being done. If we're actually concerned with
> > > interoperability and verify that things behave in the same manner between 
> > > multiple
> > > clouds then doing this would be a big step backwards. The fundamental 
> > > disconnect
> > > here is that the vendors who have implemented out of band extensions or 
> > > were
> > > taking advantage of previously available places to inject extra attributes
> > > believe that doing so means they're interoperable, which is quite far from
> > > reality. **The API is not a place for vendor differentiation.**
> > 
> > This is a temporary measure to address the fact that a large number
> > of existing tests changed their behavior, rather than having new
> > tests added to enforce this new requirement. The result is deployments
> > that previously passed these tests may no longer pass, and in fact
> > we have several cases where that's true with deployers who are
> > trying to maintain their own standard of backwards-compatibility
> > for their end users.
> 
> That's not what happened though. The API hasn't changed and the tests haven't
> really changed either. We made our enforcement on Nova's APIs a bit stricter 
> to
> ensure nothing unexpected appeared. For the most these tests work on any 
> version
> of OpenStack. (we only test it in the gate on supported stable releases, but I
> don't expect things to have drastically shifted on older releases) It also
> doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the
> only case it ever fails is when you run something extra, not from the 
> community,
> either as an extension (which themselves are going away [1]) or another 
> service
> that wraps nova or imitates nova. I'm personally not comfortable saying those
> extras are ever part of the OpenStack APIs.
>
> > We have basically three options.
> > 
> > 1. Tell deployers who are trying to do the right for their immediate
> >users that they can't use the trademark.
> > 
> > 2. Flag the related tests or remove them from the DefCore enforcement
> >suite entirely.
> > 
> > 3. Be flexible about giving consumers of Tempest time to meet the
> >new requirement by providing a way to disable the checks.
> > 
> > Option 1 goes against our own backwards compatibility policies.
> 
> I don't think backwards compatibility policies really apply to what what 
> define
> as the set of tests that as a community we are saying a vendor has to pass to
> say they're OpenStack. From my perspective as a community we either take a 
> hard
> stance on this and say to be considered an interoperable cloud (and to get the
> trademark) you have to actually have an interoperable product. We slowly 
> ratchet
> up the requirements every 6 months, there isn't any implied backwards
> compatibility in doing that. You passed in the past but not in the newer 
> stricter
> guidelines.
> 
> Also, even if I did think it applied, we're not talking about a change which
> would fall into breaking that. The change was introduced a year and half ago
> during kilo and landed a year ago during liberty:
> 
> https://review.openstack.org/#/c/156130/
> 
> That's way longer than our normal deprecation period of 3 months and a release
> boundary.
> 
> > 
> > Option 2 gives us no 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/14/2016 01:57 PM, Chris Hoge wrote:

> 
> My proposal for addressing this problem approaches it at two levels:
> 
> * For the short term, I will submit a blueprint and patch to tempest that
>   allows configuration of a grey-list of Nova APIs where strict response
>   checking on additional properties will be disabled. So, for example,
>   if the 'create  servers' API call returned extra properties on that call,
>   the strict checking on this line[8] would be disabled at runtime.
>   Use of this code path will emit a deprecation warning, and the
>   code will be scheduled for removal in 2017 directly after the release
>   of the 2017.01 guideline. Vendors would be required so submit the
>   grey-list of APIs with additional response data that would be
>   published to their marketplace entry.

To understand more. Will there be a visible asterisk with their
registration that says they require a grey-list?

> * Longer term, vendors will be expected to work with upstream to update
>   the API for returning additional data that is compatible with
>   API micro-versioning as defined by the Nova team, and the waiver would
>   no longer be allowed after the release of the 2017.01 guideline.
> 
> For the next half-year, I feel that this approach strengthens interoperability
> by accurately capturing the current state of OpenStack deployments and
> client tools. Before this change, additional properties on responses
> weren't explicitly disallowed, and vendors and deployers took advantage
> of this in production. While this is behavior that the Nova and QA teams
> want to stop, it will take a bit more time to reach downstream. Also, as
> of right now, as far as I know the only client that does strict response
> checking for Nova responses is the Tempest client. Currently, additional
> properties in responses are ignored and do not break existing client
> functionality. There is currently little to no harm done to downstream
> users by temporarily allowing additional data to be returned in responses.

In general I'm ok with this, as long as three things are true:

1) registrations that need the grey list are visually indicated quite
clearly and publicly that they needed it to pass.

2) 2017.01 is a firm cutoff.

3) We have evidence that folks that are having challenges with the
strict enforcement have made getting compliant a top priority.


3 is the one where I don't have any data either way. But I didn't see
any specs submissions (which are required for API changes in Nova) for
Newton that would indicate anyone is working on this. For 2017 to be a
hard stop, that means folks are either deleting this from their
interface, or proposing in Ocata. Which is a really short runway if this
stuff isn't super straight forward and already upstream agreed.

So I'm provisionally ok with this, if folks in the know feel like 3 is
covered.

-Sean

P.S. The Tempest changes pretty much just anticipate the Nova changes
which are deleting all these facilities in Newton -
https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/api-no-more-extensions.html
- so in some ways we aren't doing folks a ton of favors letting them
delay too far because they are about to hit a brick wall on the code side.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Tue, Jun 14, 2016 at 12:19:54PM -0700, Chris Hoge wrote:
> 
> > On Jun 14, 2016, at 11:21 AM, Matthew Treinish  wrote:
> > 
> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> >> Last year, in response to Nova micro-versioning and extension updates[1],
> >> the QA team added strict API schema checking to Tempest to ensure that
> >> no additional properties were added to Nova API responses[2][3]. In the
> >> last year, at least three vendors participating the the OpenStack Powered
> >> Trademark program have been impacted by this change, two of which
> >> reported this to the DefCore Working Group mailing list earlier this 
> >> year[4].
> >> 
> >> The DefCore Working Group determines guidelines for the OpenStack Powered
> >> program, which includes capabilities with associated functional tests
> >> from Tempest that must be passed, and designated sections with associated
> >> upstream code [5][6]. In determining these guidelines, the working group
> >> attempts to balance the future direction of development with lagging
> >> indicators of deployments and user adoption.
> >> 
> >> After a tremendous amount of consideration, I believe that the DefCore
> >> Working Group needs to implement a temporary waiver for the strict API
> >> checking requirements that were introduced last year, to give downstream
> >> deployers more time to catch up with the strict micro-versioning
> >> requirements determined by the Nova/Compute team and enforced by the
> >> Tempest/QA team.
> > 
> > I'm very much opposed to this being done. If we're actually concerned with
> > interoperability and verify that things behave in the same manner between 
> > multiple
> > clouds then doing this would be a big step backwards. The fundamental 
> > disconnect
> > here is that the vendors who have implemented out of band extensions or were
> > taking advantage of previously available places to inject extra attributes
> > believe that doing so means they're interoperable, which is quite far from
> > reality. **The API is not a place for vendor differentiation.**
> 
> Yes, it’s bad practice, but it’s also a reality, and I honestly believe that
> vendors have received the message and are working on changing.

They might be working on this, but this change was coming for quite some
time it shouldn't be a surprise to anyone at this point. I mean seriously, it's
been in tempest for 1 year, and it took 6months to land. Also, lets say we set
a hard deadline on this new option to disable the enforcement and enforce it.
Then we implement a similar change on keystone are we gonna have to do the same
thing again when vendors who have custom things running there fail.

> 
> > As a user of several clouds myself I can say that having random gorp in a
> > response makes it much more difficult to use my code against multiple 
> > clouds. I
> > have to determine which properties being returned are specific to that 
> > vendor's
> > cloud and if I actually need to depend on them for anything it makes 
> > whatever
> > code I'm writing incompatible for using against any other cloud. (unless I
> > special case that block for each cloud) Sean Dague wrote a good post where 
> > a lot
> > of this was covered a year ago when microversions was starting to pick up 
> > steam:
> > 
> > https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2 
> > 
> > 
> > I'd recommend giving it a read, he explains the user first perspective more
> > clearly there.
> > 
> > I believe Tempest in this case is doing the right thing from an 
> > interoperability
> > perspective and ensuring that the API is actually the API. Not an API with 
> > extra
> > bits a vendor decided to add.
> 
> A few points on this, though. Right now, Nova is the only API that is
> enforcing this, and the clients. While this may change in the
> future, I don’t think it accurately represents the reality of what’s
> happening in the ecosystem.

This in itself doesn't make a difference. There is a disparity in the level of
testing across all the projects. Nova happens to be further along in regards
to api stability and testing things compared to a lot of projects, it's not
really a surprise that they're the first for this to come up on. It's only a
matter of time for other projects to follow nova's example and implement similar
enforcement.

> 
> As mentioned before, we also need to balance the lagging nature of
> DefCore as an interoperability guideline with the needs of testing
> upstream changes. I’m not asking for a permanent change that
> undermines the goals of Tempest for QA, rather a temporary
> upstream modification that recognizes the challenges faced by
> vendors in the market right now, and gives them room to continue
> to align themselves with upstream. Without this, the two other 
> alternatives are to:
> 
> * Have some vendors leave the Powered program unnecessarily,
>   weakening it.
> * 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Dent]

On Tue, 14 Jun 2016, Matthew Treinish wrote:


By doing this, even as a temporary measure, we're saying it's ok to call things
an OpenStack API when you add random gorp to the responses. Which is something 
we've
very clearly said as a community is the exact opposite of the case, which the
testing reflects. I still contend just because some vendors were running old
versions of tempest and old versions of openstack where their incompatible API
changes weren't caught doesn't mean they should be given pass now.


Yes. Thanks.

--
Chris Dent   (╯°□°)╯︵┻━┻http://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 14, 2016, at 11:21 AM, Matthew Treinish  wrote:
> 
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> Last year, in response to Nova micro-versioning and extension updates[1],
>> the QA team added strict API schema checking to Tempest to ensure that
>> no additional properties were added to Nova API responses[2][3]. In the
>> last year, at least three vendors participating the the OpenStack Powered
>> Trademark program have been impacted by this change, two of which
>> reported this to the DefCore Working Group mailing list earlier this year[4].
>> 
>> The DefCore Working Group determines guidelines for the OpenStack Powered
>> program, which includes capabilities with associated functional tests
>> from Tempest that must be passed, and designated sections with associated
>> upstream code [5][6]. In determining these guidelines, the working group
>> attempts to balance the future direction of development with lagging
>> indicators of deployments and user adoption.
>> 
>> After a tremendous amount of consideration, I believe that the DefCore
>> Working Group needs to implement a temporary waiver for the strict API
>> checking requirements that were introduced last year, to give downstream
>> deployers more time to catch up with the strict micro-versioning
>> requirements determined by the Nova/Compute team and enforced by the
>> Tempest/QA team.
> 
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between 
> multiple
> clouds then doing this would be a big step backwards. The fundamental 
> disconnect
> here is that the vendors who have implemented out of band extensions or were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**

Yes, it’s bad practice, but it’s also a reality, and I honestly believe that
vendors have received the message and are working on changing.

> As a user of several clouds myself I can say that having random gorp in a
> response makes it much more difficult to use my code against multiple clouds. 
> I
> have to determine which properties being returned are specific to that 
> vendor's
> cloud and if I actually need to depend on them for anything it makes whatever
> code I'm writing incompatible for using against any other cloud. (unless I
> special case that block for each cloud) Sean Dague wrote a good post where a 
> lot
> of this was covered a year ago when microversions was starting to pick up 
> steam:
> 
> https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2 
> 
> 
> I'd recommend giving it a read, he explains the user first perspective more
> clearly there.
> 
> I believe Tempest in this case is doing the right thing from an 
> interoperability
> perspective and ensuring that the API is actually the API. Not an API with 
> extra
> bits a vendor decided to add.

A few points on this, though. Right now, Nova is the only API that is
enforcing this, and the clients. While this may change in the
future, I don’t think it accurately represents the reality of what’s
happening in the ecosystem.

As mentioned before, we also need to balance the lagging nature of
DefCore as an interoperability guideline with the needs of testing
upstream changes. I’m not asking for a permanent change that
undermines the goals of Tempest for QA, rather a temporary
upstream modification that recognizes the challenges faced by
vendors in the market right now, and gives them room to continue
to align themselves with upstream. Without this, the two other 
alternatives are to:

* Have some vendors leave the Powered program unnecessarily,
  weakening it.
* Force DefCore to adopt non-upstream testing, either as a fork
  or an independent test suite.

Neither seem ideal to me.

One of my goals is to transparently strengthen the ties between
upstream and downstream development. There is a deadline
built into this proposal, and my intention is to enforce it.

> I don't think a cloud or product that does this
> to the api should be considered an interoperable OpenStack cloud and failing 
> the
> tests is the correct behavior.

I think it’s more nuanced than this, especially right now.
Only additions to responses will be considered, not changes.
These additions will be clearly labelled as variations,
signaling the differences to users. Existing clients in use
will not break. Correct behavior will eventually be enforced,
and this would be clearly signaled by both the test tool and
through the administrative program.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > Last year, in response to Nova micro-versioning and extension updates[1],
> > > the QA team added strict API schema checking to Tempest to ensure that
> > > no additional properties were added to Nova API responses[2][3]. In the
> > > last year, at least three vendors participating the the OpenStack Powered
> > > Trademark program have been impacted by this change, two of which
> > > reported this to the DefCore Working Group mailing list earlier this 
> > > year[4].
> > > 
> > > The DefCore Working Group determines guidelines for the OpenStack Powered
> > > program, which includes capabilities with associated functional tests
> > > from Tempest that must be passed, and designated sections with associated
> > > upstream code [5][6]. In determining these guidelines, the working group
> > > attempts to balance the future direction of development with lagging
> > > indicators of deployments and user adoption.
> > > 
> > > After a tremendous amount of consideration, I believe that the DefCore
> > > Working Group needs to implement a temporary waiver for the strict API
> > > checking requirements that were introduced last year, to give downstream
> > > deployers more time to catch up with the strict micro-versioning
> > > requirements determined by the Nova/Compute team and enforced by the
> > > Tempest/QA team.
> > 
> > I'm very much opposed to this being done. If we're actually concerned with
> > interoperability and verify that things behave in the same manner between 
> > multiple
> > clouds then doing this would be a big step backwards. The fundamental 
> > disconnect
> > here is that the vendors who have implemented out of band extensions or were
> > taking advantage of previously available places to inject extra attributes
> > believe that doing so means they're interoperable, which is quite far from
> > reality. **The API is not a place for vendor differentiation.**
> 
> This is a temporary measure to address the fact that a large number
> of existing tests changed their behavior, rather than having new
> tests added to enforce this new requirement. The result is deployments
> that previously passed these tests may no longer pass, and in fact
> we have several cases where that's true with deployers who are
> trying to maintain their own standard of backwards-compatibility
> for their end users.

That's not what happened though. The API hasn't changed and the tests haven't
really changed either. We made our enforcement on Nova's APIs a bit stricter to
ensure nothing unexpected appeared. For the most these tests work on any version
of OpenStack. (we only test it in the gate on supported stable releases, but I
don't expect things to have drastically shifted on older releases) It also
doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the
only case it ever fails is when you run something extra, not from the community,
either as an extension (which themselves are going away [1]) or another service
that wraps nova or imitates nova. I'm personally not comfortable saying those
extras are ever part of the OpenStack APIs.

> 
> We have basically three options.
> 
> 1. Tell deployers who are trying to do the right for their immediate
>users that they can't use the trademark.
> 
> 2. Flag the related tests or remove them from the DefCore enforcement
>suite entirely.
> 
> 3. Be flexible about giving consumers of Tempest time to meet the
>new requirement by providing a way to disable the checks.
> 
> Option 1 goes against our own backwards compatibility policies.

I don't think backwards compatibility policies really apply to what what define
as the set of tests that as a community we are saying a vendor has to pass to
say they're OpenStack. From my perspective as a community we either take a hard
stance on this and say to be considered an interoperable cloud (and to get the
trademark) you have to actually have an interoperable product. We slowly ratchet
up the requirements every 6 months, there isn't any implied backwards
compatibility in doing that. You passed in the past but not in the newer 
stricter
guidelines.

Also, even if I did think it applied, we're not talking about a change which
would fall into breaking that. The change was introduced a year and half ago
during kilo and landed a year ago during liberty:

https://review.openstack.org/#/c/156130/

That's way longer than our normal deprecation period of 3 months and a release
boundary.

> 
> Option 2 gives us no winners and actually reduces the interoperability
> guarantees we already have in place.
> 
> Option 3 applies our usual community standard of slowly rolling
> forward while maintaining compatibility as broadly as possible.

Except in this case there isn't actually any compatibility being maintained.
We're 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > Last year, in response to Nova micro-versioning and extension updates[1],
> > the QA team added strict API schema checking to Tempest to ensure that
> > no additional properties were added to Nova API responses[2][3]. In the
> > last year, at least three vendors participating the the OpenStack Powered
> > Trademark program have been impacted by this change, two of which
> > reported this to the DefCore Working Group mailing list earlier this 
> > year[4].
> > 
> > The DefCore Working Group determines guidelines for the OpenStack Powered
> > program, which includes capabilities with associated functional tests
> > from Tempest that must be passed, and designated sections with associated
> > upstream code [5][6]. In determining these guidelines, the working group
> > attempts to balance the future direction of development with lagging
> > indicators of deployments and user adoption.
> > 
> > After a tremendous amount of consideration, I believe that the DefCore
> > Working Group needs to implement a temporary waiver for the strict API
> > checking requirements that were introduced last year, to give downstream
> > deployers more time to catch up with the strict micro-versioning
> > requirements determined by the Nova/Compute team and enforced by the
> > Tempest/QA team.
> 
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between 
> multiple
> clouds then doing this would be a big step backwards. The fundamental 
> disconnect
> here is that the vendors who have implemented out of band extensions or were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**

This is a temporary measure to address the fact that a large number
of existing tests changed their behavior, rather than having new
tests added to enforce this new requirement. The result is deployments
that previously passed these tests may no longer pass, and in fact
we have several cases where that's true with deployers who are
trying to maintain their own standard of backwards-compatibility
for their end users.

We have basically three options.

1. Tell deployers who are trying to do the right for their immediate
   users that they can't use the trademark.

2. Flag the related tests or remove them from the DefCore enforcement
   suite entirely.

3. Be flexible about giving consumers of Tempest time to meet the
   new requirement by providing a way to disable the checks.

Option 1 goes against our own backwards compatibility policies.

Option 2 gives us no winners and actually reduces the interoperability
guarantees we already have in place.

Option 3 applies our usual community standard of slowly rolling
forward while maintaining compatibility as broadly as possible.

No one is suggesting that a permanent, or even open-ended, exception
be granted.

Doug

> 
> As a user of several clouds myself I can say that having random gorp in a
> response makes it much more difficult to use my code against multiple clouds. 
> I
> have to determine which properties being returned are specific to that 
> vendor's
> cloud and if I actually need to depend on them for anything it makes whatever
> code I'm writing incompatible for using against any other cloud. (unless I
> special case that block for each cloud) Sean Dague wrote a good post where a 
> lot
> of this was covered a year ago when microversions was starting to pick up 
> steam:
> 
> https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2
> 
> I'd recommend giving it a read, he explains the user first perspective more
> clearly there.
> 
> I believe Tempest in this case is doing the right thing from an 
> interoperability
> perspective and ensuring that the API is actually the API. Not an API with 
> extra
> bits a vendor decided to add. I don't think a cloud or product that does this
> to the api should be considered an interoperable OpenStack cloud and failing 
> the
> tests is the correct behavior.
> 
> -Matt Treinish
> 
> > 
> > My reasoning behind this is that while the change that enabled strict
> > checking was discussed publicly in the developer community and took
> > some time to be implemented, it still landed quickly and broke several
> > existing deployments overnight. As Tempest has moved forward with
> > bug and UX fixes (some in part to support the interoperability testing
> > efforts of the DefCore Working Group), using an older versions of Tempest
> > where this strict checking is not enforced is no longer a viable solution
> > for downstream deployers. The TC has passed a resolution to advise
> > DefCore to use Tempest as the single source of capability testing[7],
> > but this naturally 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> Last year, in response to Nova micro-versioning and extension updates[1],
> the QA team added strict API schema checking to Tempest to ensure that
> no additional properties were added to Nova API responses[2][3]. In the
> last year, at least three vendors participating the the OpenStack Powered
> Trademark program have been impacted by this change, two of which
> reported this to the DefCore Working Group mailing list earlier this year[4].
> 
> The DefCore Working Group determines guidelines for the OpenStack Powered
> program, which includes capabilities with associated functional tests
> from Tempest that must be passed, and designated sections with associated
> upstream code [5][6]. In determining these guidelines, the working group
> attempts to balance the future direction of development with lagging
> indicators of deployments and user adoption.
> 
> After a tremendous amount of consideration, I believe that the DefCore
> Working Group needs to implement a temporary waiver for the strict API
> checking requirements that were introduced last year, to give downstream
> deployers more time to catch up with the strict micro-versioning
> requirements determined by the Nova/Compute team and enforced by the
> Tempest/QA team.

I'm very much opposed to this being done. If we're actually concerned with
interoperability and verify that things behave in the same manner between 
multiple
clouds then doing this would be a big step backwards. The fundamental disconnect
here is that the vendors who have implemented out of band extensions or were
taking advantage of previously available places to inject extra attributes
believe that doing so means they're interoperable, which is quite far from
reality. **The API is not a place for vendor differentiation.**

As a user of several clouds myself I can say that having random gorp in a
response makes it much more difficult to use my code against multiple clouds. I
have to determine which properties being returned are specific to that vendor's
cloud and if I actually need to depend on them for anything it makes whatever
code I'm writing incompatible for using against any other cloud. (unless I
special case that block for each cloud) Sean Dague wrote a good post where a lot
of this was covered a year ago when microversions was starting to pick up steam:

https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2

I'd recommend giving it a read, he explains the user first perspective more
clearly there.

I believe Tempest in this case is doing the right thing from an interoperability
perspective and ensuring that the API is actually the API. Not an API with extra
bits a vendor decided to add. I don't think a cloud or product that does this
to the api should be considered an interoperable OpenStack cloud and failing the
tests is the correct behavior.

-Matt Treinish

> 
> My reasoning behind this is that while the change that enabled strict
> checking was discussed publicly in the developer community and took
> some time to be implemented, it still landed quickly and broke several
> existing deployments overnight. As Tempest has moved forward with
> bug and UX fixes (some in part to support the interoperability testing
> efforts of the DefCore Working Group), using an older versions of Tempest
> where this strict checking is not enforced is no longer a viable solution
> for downstream deployers. The TC has passed a resolution to advise
> DefCore to use Tempest as the single source of capability testing[7],
> but this naturally introduces tension between the competing goals of
> maintaining upstream functional testing and also tracking lagging
> indicators.
> 
> My proposal for addressing this problem approaches it at two levels:
> 
> * For the short term, I will submit a blueprint and patch to tempest that
>   allows configuration of a grey-list of Nova APIs where strict response
>   checking on additional properties will be disabled. So, for example,
>   if the 'create  servers' API call returned extra properties on that call,
>   the strict checking on this line[8] would be disabled at runtime.
>   Use of this code path will emit a deprecation warning, and the
>   code will be scheduled for removal in 2017 directly after the release
>   of the 2017.01 guideline. Vendors would be required so submit the
>   grey-list of APIs with additional response data that would be
>   published to their marketplace entry.
> 
> * Longer term, vendors will be expected to work with upstream to update
>   the API for returning additional data that is compatible with
>   API micro-versioning as defined by the Nova team, and the waiver would
>   no longer be allowed after the release of the 2017.01 guideline.
> 
> For the next half-year, I feel that this approach strengthens interoperability
> by accurately capturing the current state of OpenStack deployments and
> client tools. Before this change, additional properties on 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Chris Hoge's message of 2016-06-14 10:57:05 -0700:
> Last year, in response to Nova micro-versioning and extension updates[1],
> the QA team added strict API schema checking to Tempest to ensure that
> no additional properties were added to Nova API responses[2][3]. In the
> last year, at least three vendors participating the the OpenStack Powered
> Trademark program have been impacted by this change, two of which
> reported this to the DefCore Working Group mailing list earlier this year[4].
> 
> The DefCore Working Group determines guidelines for the OpenStack Powered
> program, which includes capabilities with associated functional tests
> from Tempest that must be passed, and designated sections with associated
> upstream code [5][6]. In determining these guidelines, the working group
> attempts to balance the future direction of development with lagging
> indicators of deployments and user adoption.
> 
> After a tremendous amount of consideration, I believe that the DefCore
> Working Group needs to implement a temporary waiver for the strict API
> checking requirements that were introduced last year, to give downstream
> deployers more time to catch up with the strict micro-versioning
> requirements determined by the Nova/Compute team and enforced by the
> Tempest/QA team.
> 
> My reasoning behind this is that while the change that enabled strict
> checking was discussed publicly in the developer community and took
> some time to be implemented, it still landed quickly and broke several
> existing deployments overnight. As Tempest has moved forward with
> bug and UX fixes (some in part to support the interoperability testing
> efforts of the DefCore Working Group), using an older versions of Tempest
> where this strict checking is not enforced is no longer a viable solution
> for downstream deployers. The TC has passed a resolution to advise
> DefCore to use Tempest as the single source of capability testing[7],
> but this naturally introduces tension between the competing goals of
> maintaining upstream functional testing and also tracking lagging
> indicators.
> 
> My proposal for addressing this problem approaches it at two levels:
> 
> * For the short term, I will submit a blueprint and patch to tempest that
>   allows configuration of a grey-list of Nova APIs where strict response
>   checking on additional properties will be disabled. So, for example,
>   if the 'create  servers' API call returned extra properties on that call,
>   the strict checking on this line[8] would be disabled at runtime.
>   Use of this code path will emit a deprecation warning, and the
>   code will be scheduled for removal in 2017 directly after the release
>   of the 2017.01 guideline. Vendors would be required so submit the
>   grey-list of APIs with additional response data that would be
>   published to their marketplace entry.
> 
> * Longer term, vendors will be expected to work with upstream to update
>   the API for returning additional data that is compatible with
>   API micro-versioning as defined by the Nova team, and the waiver would
>   no longer be allowed after the release of the 2017.01 guideline.
> 
> For the next half-year, I feel that this approach strengthens interoperability
> by accurately capturing the current state of OpenStack deployments and
> client tools. Before this change, additional properties on responses
> weren't explicitly disallowed, and vendors and deployers took advantage
> of this in production. While this is behavior that the Nova and QA teams
> want to stop, it will take a bit more time to reach downstream. Also, as
> of right now, as far as I know the only client that does strict response
> checking for Nova responses is the Tempest client. Currently, additional
> properties in responses are ignored and do not break existing client
> functionality. There is currently little to no harm done to downstream
> users by temporarily allowing additional data to be returned in responses.

Thanks for putting this proposal together, Chris. The configuration
option you describe makes sense as a temporary solution to the
issue, and the timeline you propose (combined with the past year
since the change went in) should be plenty of time to handle upgrades.

Doug

> 
> Thanks,
> 
> Chris Hoge
> Interop Engineer
> OpenStack Foundation
> 
> [1] 
> https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
> [2] 
> http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
> [3] https://review.openstack.org/#/c/156130
> [4] 
> http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
> [5] http://git.openstack.org/cgit/openstack/defcore/tree/2015.07.json
> [6] http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json
> [7] 
> http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst
> [8] 
> 

[openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

Last year, in response to Nova micro-versioning and extension updates[1],
the QA team added strict API schema checking to Tempest to ensure that
no additional properties were added to Nova API responses[2][3]. In the
last year, at least three vendors participating the the OpenStack Powered
Trademark program have been impacted by this change, two of which
reported this to the DefCore Working Group mailing list earlier this year[4].

The DefCore Working Group determines guidelines for the OpenStack Powered
program, which includes capabilities with associated functional tests
from Tempest that must be passed, and designated sections with associated
upstream code [5][6]. In determining these guidelines, the working group
attempts to balance the future direction of development with lagging
indicators of deployments and user adoption.

After a tremendous amount of consideration, I believe that the DefCore
Working Group needs to implement a temporary waiver for the strict API
checking requirements that were introduced last year, to give downstream
deployers more time to catch up with the strict micro-versioning
requirements determined by the Nova/Compute team and enforced by the
Tempest/QA team.

My reasoning behind this is that while the change that enabled strict
checking was discussed publicly in the developer community and took
some time to be implemented, it still landed quickly and broke several
existing deployments overnight. As Tempest has moved forward with
bug and UX fixes (some in part to support the interoperability testing
efforts of the DefCore Working Group), using an older versions of Tempest
where this strict checking is not enforced is no longer a viable solution
for downstream deployers. The TC has passed a resolution to advise
DefCore to use Tempest as the single source of capability testing[7],
but this naturally introduces tension between the competing goals of
maintaining upstream functional testing and also tracking lagging
indicators.

My proposal for addressing this problem approaches it at two levels:

* For the short term, I will submit a blueprint and patch to tempest that
  allows configuration of a grey-list of Nova APIs where strict response
  checking on additional properties will be disabled. So, for example,
  if the 'create  servers' API call returned extra properties on that call,
  the strict checking on this line[8] would be disabled at runtime.
  Use of this code path will emit a deprecation warning, and the
  code will be scheduled for removal in 2017 directly after the release
  of the 2017.01 guideline. Vendors would be required so submit the
  grey-list of APIs with additional response data that would be
  published to their marketplace entry.

* Longer term, vendors will be expected to work with upstream to update
  the API for returning additional data that is compatible with
  API micro-versioning as defined by the Nova team, and the waiver would
  no longer be allowed after the release of the 2017.01 guideline.

For the next half-year, I feel that this approach strengthens interoperability
by accurately capturing the current state of OpenStack deployments and
client tools. Before this change, additional properties on responses
weren't explicitly disallowed, and vendors and deployers took advantage
of this in production. While this is behavior that the Nova and QA teams
want to stop, it will take a bit more time to reach downstream. Also, as
of right now, as far as I know the only client that does strict response
checking for Nova responses is the Tempest client. Currently, additional
properties in responses are ignored and do not break existing client
functionality. There is currently little to no harm done to downstream
users by temporarily allowing additional data to be returned in responses.

Thanks,

Chris Hoge
Interop Engineer
OpenStack Foundation

[1] 
https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
[3] https://review.openstack.org/#/c/156130
[4] 
http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
[5] http://git.openstack.org/cgit/openstack/defcore/tree/2015.07.json
[6] http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json
[7] 
http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst
[8] 
http://git.openstack.org/cgit/openstack/tempest-lib/tree/tempest_lib/api_schema/response/compute/v2_1/servers.py#n39


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev