Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

(not for usage questions) Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing > On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote: > > Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +: > >> On Tue, Jun

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[mailto:ch...@openstack.org] Sent: Wednesday, June 22, 2016 1:37 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing > On Jun 22, 2016, at 11:24 AM, Sean Dague wrote:

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/22/2016 03:13 PM, Jay Faulkner wrote: > > > On 6/22/16 12:01 PM, Sean Dague wrote: >> On 06/22/2016 02:37 PM, Chris Hoge wrote: On Jun 22, 2016, at 11:24 AM, Sean Dague wrote: On 06/22/2016 01:59 PM, Chris Hoge wrote: >> On Jun 20, 2016, at 5:10 AM, Sean

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 6/22/16 12:01 PM, Sean Dague wrote: > On 06/22/2016 02:37 PM, Chris Hoge wrote: >>> On Jun 22, 2016, at 11:24 AM, Sean Dague wrote: >>> >>> On 06/22/2016 01:59 PM, Chris Hoge wrote: > On Jun 20, 2016, at 5:10 AM, Sean Dague >

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/22/2016 02:37 PM, Chris Hoge wrote: > >> On Jun 22, 2016, at 11:24 AM, Sean Dague wrote: >> >> On 06/22/2016 01:59 PM, Chris Hoge wrote: >>> On Jun 20, 2016, at 5:10 AM, Sean Dague > wrote: On 06/14/2016 07:19 PM,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 22, 2016, at 11:24 AM, Sean Dague wrote: > > On 06/22/2016 01:59 PM, Chris Hoge wrote: >> >>> On Jun 20, 2016, at 5:10 AM, Sean Dague >> > wrote: >>> >>> On 06/14/2016 07:19 PM, Chris Hoge wrote: > On Jun 14, 2016, at

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/22/2016 01:59 PM, Chris Hoge wrote: > >> On Jun 20, 2016, at 5:10 AM, Sean Dague > > wrote: >> >> On 06/14/2016 07:19 PM, Chris Hoge wrote: >>> On Jun 14, 2016, at 3:59 PM, Edward Leafe > wrote:

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 20, 2016, at 5:10 AM, Sean Dague wrote: > > On 06/14/2016 07:19 PM, Chris Hoge wrote: >> >>> On Jun 14, 2016, at 3:59 PM, Edward Leafe wrote: >>> >>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish wrote: >>> But, if we add

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 20, 2016, at 6:56 AM, Doug Hellmann wrote: > > > I'm also, I think, edging away from the "we need to find a compromise" > camp into the "why is this turning into such a big deal" camp. How did > we get into a situation where the community has set a clear

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Mark Voelker's message of 2016-06-20 13:24:46 +: > > > On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote: > > > > Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +: > > > >> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote: >

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote: > > Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +: > >> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote: >> >> >>> I don't think DefCore actually needs to change old versions of

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +: > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote: > > > > I don't think DefCore actually needs to change old versions of Tempest, > > but maybe Chris or Mark can verify that? > > So if I’m groking this correctly,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/14/2016 07:19 PM, Chris Hoge wrote: > >> On Jun 14, 2016, at 3:59 PM, Edward Leafe wrote: >> >> On Jun 14, 2016, at 5:50 PM, Matthew Treinish wrote: >> >>> But, if we add another possible state on the defcore side like conditional >>> pass, >>>

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

2016-06-16 2:26 GMT-07:00 Morgan Fainberg : > On Wed, Jun 15, 2016 at 11:54 PM, Ken'ichi Ohmichi > wrote: >> >> This discussion was expected when we implemented the Tempest patch, >> then I sent a mail to defcore comittee[1] >> As the above ml, "A

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 6/14/2016 6:37 PM, Chris Hoge wrote: I’m beginning to wonder if we need to make DefCore use release branches then back-port bug-fixes and relevant features additions as necessary. To clarify, do you mean use release branches from the upstream repos? Because if DefCore is supporting 2

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 6/14/2016 6:19 PM, Chris Hoge wrote: One would hope that micro-versions would be able to address this exact issue for vendors by giving them a means to propose optional but well-defined API response additions (not extensions) that are defined upstream and usable by all vendors. If it’s not

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 23:53 Jun 17, Matthew Treinish wrote: > On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote: > > On 15:12 Jun 14, Matthew Treinish wrote: > > > I don't think backwards compatibility policies really apply to what what > > > define > > > as the set of tests that as a community we are

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote: > On 15:12 Jun 14, Matthew Treinish wrote: > > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 15:12 Jun 14, Matthew Treinish wrote: > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > > We have basically three options. > > > > 1.

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 15 June 2016 at 02:37, Sean Dague wrote: > On 06/14/2016 07:28 PM, Monty Taylor wrote: >> >> On 06/14/2016 05:42 PM, Doug Hellmann wrote: > > >> >> I think this is the most important thing to me as it relates to this. >> I'm obviously a huge proponent of clouds behaving more

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Jun 16, 2016, at 2:25 PM, Matthew Treinish wrote: On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote: Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400: On Thu, Jun 16, 2016 at 12:59:41PM

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400: > > On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote: > > > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400: > > > > On Wed,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400: > On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote: > > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400: > > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: > > > > Excerpts

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400: > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: > > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: > > > > Top posting one

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400: > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: > > > Top posting one note and direct comments inline, I’m proposing > > > this as a member

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 16/06/2016 00:30, Matthew Treinish wrote: > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: >> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: >>> Top posting one note and direct comments inline, I’m proposing >>> this as a member of the DefCore working group,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Wed, Jun 15, 2016 at 11:54 PM, Ken'ichi Ohmichi wrote: > This discussion was expected when we implemented the Tempest patch, > then I sent a mail to defcore comittee[1] > As the above ml, "A DefCore Guideline typically covers three OpenStack > releases". > That means

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

This discussion was expected when we implemented the Tempest patch, then I sent a mail to defcore comittee[1] As the above ml, "A DefCore Guideline typically covers three OpenStack releases". That means the latest guideline needs to cover Mitaka, Liberty and Kilo, right? In the Kilo development,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Jun 14, 2016 14:42, "Doug Hellmann" wrote: > > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: > > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > > >

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> -Original Message- > From: GHANSHYAM MANN [mailto:ghanshyamm...@gmail.com] > Sent: Wednesday, June 15, 2016 9:59 PM > To: OpenStack Development Mailing List (not for usage questions) > <openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [tempest][n

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Wed, Jun 15, 2016 at 6:12 AM, Matthew Treinish wrote: > On Tue, Jun 14, 2016 at 12:19:54PM -0700, Chris Hoge wrote: >> >> > On Jun 14, 2016, at 11:21 AM, Matthew Treinish >> > wrote: >> > >> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: > > Top posting one note and direct comments inline, I’m proposing > > this as a member of the DefCore working group, but this > > proposal itself has not been accepted

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 15, 2016, at 8:01 AM, Sean Dague wrote: > > On 06/15/2016 12:14 AM, Mark Voelker wrote: > >> >> It is perhaps important to note here that the DefCore seems to have two >> meanings to a lot of people I talk to today: it’s a mark of interoperability >> (the OpenStack

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: > Top posting one note and direct comments inline, I’m proposing > this as a member of the DefCore working group, but this > proposal itself has not been accepted as the forward course of > action by the working group. These are my

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/15/2016 12:14 AM, Mark Voelker wrote: > > It is perhaps important to note here that the DefCore seems to have two > meanings to a lot of people I talk to today: it’s a mark of interoperability > (the OpenStack Powered badge that says certain capabilities of this cloud > behave like

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Sean Dague wrote: On 06/14/2016 07:28 PM, Monty Taylor wrote: On 06/14/2016 05:42 PM, Doug Hellmann wrote: I think this is the most important thing to me as it relates to this. I'm obviously a huge proponent of clouds behaving more samely. But I also think that, as Doug nicely describes

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Jun 14, 2016, at 7:28 PM, Monty Taylor wrote: > > On 06/14/2016 05:42 PM, Doug Hellmann wrote: >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: Excerpts from Matthew

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/14/2016 07:28 PM, Monty Taylor wrote: On 06/14/2016 05:42 PM, Doug Hellmann wrote: I think this is the most important thing to me as it relates to this. I'm obviously a huge proponent of clouds behaving more samely. But I also think that, as Doug nicely describes above, we've sort of

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Chris Hoge's message of 2016-06-14 16:19:11 -0700: > > > On Jun 14, 2016, at 3:59 PM, Edward Leafe wrote: > > > > On Jun 14, 2016, at 5:50 PM, Matthew Treinish wrote: > > > >> But, if we add another possible state on the defcore side like

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Top posting one note and direct comments inline, I’m proposing this as a member of the DefCore working group, but this proposal itself has not been accepted as the forward course of action by the working group. These are my own views as the administrator of the program and not that of the working

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/14/2016 05:42 PM, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: >> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: >>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: On Tue, Jun 14, 2016 at 10:57:05AM

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 14, 2016, at 3:59 PM, Edward Leafe wrote: > > On Jun 14, 2016, at 5:50 PM, Matthew Treinish wrote: > >> But, if we add another possible state on the defcore side like conditional >> pass, >> warning, yellow, etc. (the name doesn't matter) which

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Jun 14, 2016, at 5:50 PM, Matthew Treinish wrote: > But, if we add another possible state on the defcore side like conditional > pass, > warning, yellow, etc. (the name doesn't matter) which is used to indicate that > things on product X could only pass when strict

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: > > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > > > On Tue,

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Sean Dague's message of 2016-06-14 17:29:06 -0400: > On 06/14/2016 01:57 PM, Chris Hoge wrote: > > > > > My proposal for addressing this problem approaches it at two levels: > > > > * For the short term, I will submit a blueprint and patch to tempest that > > allows

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > > > > Last year, in

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On 06/14/2016 01:57 PM, Chris Hoge wrote: > > My proposal for addressing this problem approaches it at two levels: > > * For the short term, I will submit a blueprint and patch to tempest that > allows configuration of a grey-list of Nova APIs where strict response > checking on additional

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Tue, Jun 14, 2016 at 12:19:54PM -0700, Chris Hoge wrote: > > > On Jun 14, 2016, at 11:21 AM, Matthew Treinish wrote: > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > >> Last year, in response to Nova micro-versioning and extension updates[1], > >>

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Tue, 14 Jun 2016, Matthew Treinish wrote: By doing this, even as a temporary measure, we're saying it's ok to call things an OpenStack API when you add random gorp to the responses. Which is something we've very clearly said as a community is the exact opposite of the case, which the

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

> On Jun 14, 2016, at 11:21 AM, Matthew Treinish wrote: > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: >> Last year, in response to Nova micro-versioning and extension updates[1], >> the QA team added strict API schema checking to Tempest to ensure that >>

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > > > Last year, in response to Nova micro-versioning and extension updates[1], > > > the QA team

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > > Last year, in response to Nova micro-versioning and extension updates[1], > > the QA team added strict API schema checking to Tempest to ensure that > > no

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote: > Last year, in response to Nova micro-versioning and extension updates[1], > the QA team added strict API schema checking to Tempest to ensure that > no additional properties were added to Nova API responses[2][3]. In the > last year, at

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Excerpts from Chris Hoge's message of 2016-06-14 10:57:05 -0700: > Last year, in response to Nova micro-versioning and extension updates[1], > the QA team added strict API schema checking to Tempest to ensure that > no additional properties were added to Nova API responses[2][3]. In the > last

[openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Last year, in response to Nova micro-versioning and extension updates[1], the QA team added strict API schema checking to Tempest to ensure that no additional properties were added to Nova API responses[2][3]. In the last year, at least three vendors participating the the OpenStack Powered