Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Matt Riedemann]

On 9/9/2018 1:11 PM, Ghanshyam Mann wrote:

Yeah, Tempest would not fit as best location for such tagging or whitelist. I 
think nova may be better choice if nothing else.


OK I've thrown this on the nova ptg etherpad agenda [1] for a misc item 
to talk about.


[1] https://etherpad.openstack.org/p/nova-ptg-stein

--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Ghanshyam Mann]

  On Sat, 08 Sep 2018 08:28:06 +0900 Matt Riedemann  
wrote  
 > On 9/7/2018 10:25 AM, William M Edmonds wrote:
 > > The concern that I have with whitelisting in a given CI is that it has 
 > > to be done independently in every compute driver CI. So while I agree 
 > > that it won't be easy to maintain tagging for compute driver on the 
 > > tempest side, at least that's one place / easier than doing it in every 
 > > driver CI. When anyone figures out that an change is needed, all of the 
 > > CIs would benefit together if there is a shared solution.
 > 
 > How about storing the compute-driver specific whitelist in a common 
 > location? I'm not sure if that would be tempest, nova or somewhere else.

Yeah, Tempest would not fit as best location for such tagging or whitelist. I 
think nova may be better choice if nothing else.

 > 
 > -- 
 > 
 > Thanks,
 > 
 > Matt
 > 
 > __
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 > 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Matt Riedemann]

On 9/7/2018 10:25 AM, William M Edmonds wrote:
The concern that I have with whitelisting in a given CI is that it has 
to be done independently in every compute driver CI. So while I agree 
that it won't be easy to maintain tagging for compute driver on the 
tempest side, at least that's one place / easier than doing it in every 
driver CI. When anyone figures out that an change is needed, all of the 
CIs would benefit together if there is a shared solution.


How about storing the compute-driver specific whitelist in a common 
location? I'm not sure if that would be tempest, nova or somewhere else.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[William M Edmonds]

Ghanshyam Mann  wrote on 09/07/2018 02:18:13 AM:

snip..

> neutron-tempest-plugin or other service test you can always avoid to
> run with regex. And i do not think compute negative or DB test will
> take much time to run. But still if you want to avoid to run then, I
> think it is easy to maintain a whitelist regex file on CI side which
> can run only compute driver tests(61 in this case).
>
> Tagging compute driver on tempest side is little hard to maintain i
> feel and it can goes out of date very easily. If you have any other
> idea on that, we can surly talk in PTG on this.

The concern that I have with whitelisting in a given CI is that it has to
be done independently in every compute driver CI. So while I agree that it
won't be easy to maintain tagging for compute driver on the tempest side,
at least that's one place / easier than doing it in every driver CI. When
anyone figures out that an change is needed, all of the CIs would benefit
together if there is a shared solution.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Chen CH Ji]

Thanks for the confirmation, I agree use internal maintain CI whitelist is
a good way, and I confirmed with our CI folks that we already did so and
more can be removed per
Eric's test result below, so we will compare and remove unnecessary cases
to get more bandwidth to run 3rd party CI. thanks

Best Regards!

Kevin (Chen) Ji 纪 晨

Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN   Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC



From:   Ghanshyam Mann 
To: "OpenStack Development Mailing List \\"

Date:   09/07/2018 02:18 PM
Subject:    Re: [openstack-dev] [tempest][CI][nova compute] Skipping
non-compute-driver tests






  On Fri, 07 Sep 2018 04:41:32 +0900 Eric Fried 
wrote 
 > Jichen-
 >
 > That patch is not ever intended to merge; hope that was clear from
the
 > start :) It was just a proving ground to demonstrate which tests still
 > pass when there's effectively no compute driver in play.
 >
 > We haven't taken any action on this from our end, though we have
done a
 > little brainstorming about how we would tool our CI to skip those tests
 > most (but not all) of the time. Happy to share our experiences with you
 > if/as we move forward with that.
 >
 > Regarding the tempest-level automation, I certainly had z in mind
when
 > I was thinking about it. If you have the time and inclination to help
 > look into it, that would be most welcome.

Sorry for late response, somehow i missed this thread. As you mentioned and
noticed in your patch that there are ~700 tests which does not touch
compute driver. Most of them are from neutron-tempest-plugins or other
service tests. From Tempest compute tests, many of them are negative tests
or DB layer tests [1].

neutron-tempest-plugin or other service test you can always avoid to run
with regex. And i do not think compute negative or DB test will take much
time to run. But still if you want to avoid to run then, I think it is easy
to maintain a whitelist regex file on CI side which can run only compute
driver tests(61 in this case).

Tagging compute driver on tempest side is little hard to maintain i feel
and it can goes out of date very easily. If you have any other idea on
that, we can surly talk in PTG on this.

[1]
http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz


 >
 > Thanks,
 > efried
 >
 > On 09/06/2018 12:33 AM, Chen CH Ji wrote:
 > > I see the patch is still ongoing status and do you have a follow up
 > > plan/discussion for that? we are maintaining 2 CIs (z/VM and KVM on z)
 > > so skip non-compute related cases will be a good for 3rd part CI ..
thanks
 > >
 > > Best Regards!
 > >
 > > Kevin (Chen) Ji 纪 晨
 > >
 > > Engineer, zVM Development, CSTL
 > > Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
 > > Phone: +86-10-82451493
 > > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
 > > District, Beijing 100193, PRC
 > >
 > > Inactive hide details for Eric Fried ---09/04/2018 09:35:09
PM---Folks-
 > > The other day, I posted an experimental patch [1] withEric Fried
 > > ---09/04/2018 09:35:09 PM---Folks- The other day, I posted an
 > > experimental patch [1] with an effectively
 > >
 > > From: Eric Fried 
 > > To: "OpenStack Development Mailing List (not for usage questions)"
 > > 
 > > Date: 09/04/2018 09:35 PM
 > > Subject: [openstack-dev] [tempest][CI][nova compute] Skipping
 > > non-compute-driver tests
 > >
 > >

 > >
 > >
 > >
 > > Folks-
 > >
 > > The other day, I posted an experimental patch [1] with an effectively
 > > empty ComputeDriver (just enough to make n-cpu actually start) to see
 > > how much of our CI would pass. The theory being that any tests that
 > > still pass are tests that don't touch our compute driver, and are
 > > therefore not useful to run in our CI environment. Because anything
that
 > > doesn't touch our code should already be well covered by generic
 > > dsvm-tempest CIs. The results [2] show that 707 tests still pass.
 > >
 > > So I'm wondering whether there might be a way to mark tests as being
 > > "compute driver-specific" such that we could switch off all the other
 > > ones [3] via a one-line conf setting. Because surely this has
potential
 > > to save a lot of CI resource not just for us but for other driver
 > > vendors, in tree and out.
 > >
 > > Thanks,
 > > efried
 > >
 > > [1]
http

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Ghanshyam Mann]

  On Fri, 07 Sep 2018 04:41:32 +0900 Eric Fried  wrote 
 
 > Jichen-
 > 
 > That patch is not ever intended to merge; hope that was clear from the
 > start :) It was just a proving ground to demonstrate which tests still
 > pass when there's effectively no compute driver in play.
 > 
 > We haven't taken any action on this from our end, though we have done a
 > little brainstorming about how we would tool our CI to skip those tests
 > most (but not all) of the time. Happy to share our experiences with you
 > if/as we move forward with that.
 > 
 > Regarding the tempest-level automation, I certainly had z in mind when
 > I was thinking about it. If you have the time and inclination to help
 > look into it, that would be most welcome.

Sorry for late response, somehow i missed this thread. As you mentioned and 
noticed in your patch that there are ~700 tests which does not touch compute 
driver. Most of them are from neutron-tempest-plugins or other service tests. 
From Tempest compute tests, many of them are negative tests or DB layer tests 
[1].

neutron-tempest-plugin or other service test you can always avoid to run with 
regex. And i do not think compute negative or DB test will take much time to 
run. But still if you want to avoid to run then, I think it is easy to maintain 
a whitelist regex file on CI side which can run only compute driver tests(61 in 
this case). 

Tagging compute driver on tempest side is little hard to maintain i feel and it 
can goes out of date very easily. If you have any other idea on that, we can 
surly talk in PTG on this. 

[1] 
http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz

 > 
 > Thanks,
 > efried
 > 
 > On 09/06/2018 12:33 AM, Chen CH Ji wrote:
 > > I see the patch is still ongoing status and do you have a follow up
 > > plan/discussion for that? we are maintaining 2 CIs (z/VM and KVM on z)
 > > so skip non-compute related cases will be a good for 3rd part CI .. thanks
 > > 
 > > Best Regards!
 > > 
 > > Kevin (Chen) Ji 纪 晨
 > > 
 > > Engineer, zVM Development, CSTL
 > > Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
 > > Phone: +86-10-82451493
 > > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
 > > District, Beijing 100193, PRC
 > > 
 > > Inactive hide details for Eric Fried ---09/04/2018 09:35:09 PM---Folks-
 > > The other day, I posted an experimental patch [1] withEric Fried
 > > ---09/04/2018 09:35:09 PM---Folks- The other day, I posted an
 > > experimental patch [1] with an effectively
 > > 
 > > From: Eric Fried 
 > > To: "OpenStack Development Mailing List (not for usage questions)"
 > > 
 > > Date: 09/04/2018 09:35 PM
 > > Subject: [openstack-dev] [tempest][CI][nova compute] Skipping
 > > non-compute-driver tests
 > > 
 > > 
 > > 
 > > 
 > > 
 > > Folks-
 > > 
 > > The other day, I posted an experimental patch [1] with an effectively
 > > empty ComputeDriver (just enough to make n-cpu actually start) to see
 > > how much of our CI would pass. The theory being that any tests that
 > > still pass are tests that don't touch our compute driver, and are
 > > therefore not useful to run in our CI environment. Because anything that
 > > doesn't touch our code should already be well covered by generic
 > > dsvm-tempest CIs. The results [2] show that 707 tests still pass.
 > > 
 > > So I'm wondering whether there might be a way to mark tests as being
 > > "compute driver-specific" such that we could switch off all the other
 > > ones [3] via a one-line conf setting. Because surely this has potential
 > > to save a lot of CI resource not just for us but for other driver
 > > vendors, in tree and out.
 > > 
 > > Thanks,
 > > efried
 > > 
 > > [1] https://review.openstack.org/#/c/599066/
 > > [2]
 > > http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz
 > > [3] I get that there's still value in running all those tests. But it
 > > could be done like once every 10 or 50 or 100 runs instead of every time.
 > > 
 > > __
 > > OpenStack Development Mailing List (not for usage questions)
 > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 > > 
 > > 
 > > 
 > > 
 > > 
 > > 
 > >

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Eric Fried]

Jichen-

That patch is not ever intended to merge; hope that was clear from the
start :) It was just a proving ground to demonstrate which tests still
pass when there's effectively no compute driver in play.

We haven't taken any action on this from our end, though we have done a
little brainstorming about how we would tool our CI to skip those tests
most (but not all) of the time. Happy to share our experiences with you
if/as we move forward with that.

Regarding the tempest-level automation, I certainly had z in mind when
I was thinking about it. If you have the time and inclination to help
look into it, that would be most welcome.

Thanks,
efried

On 09/06/2018 12:33 AM, Chen CH Ji wrote:
> I see the patch is still ongoing status and do you have a follow up
> plan/discussion for that? we are maintaining 2 CIs (z/VM and KVM on z)
> so skip non-compute related cases will be a good for 3rd part CI .. thanks
> 
> Best Regards!
> 
> Kevin (Chen) Ji 纪 晨
> 
> Engineer, zVM Development, CSTL
> Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
> Phone: +86-10-82451493
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> Inactive hide details for Eric Fried ---09/04/2018 09:35:09 PM---Folks-
> The other day, I posted an experimental patch [1] withEric Fried
> ---09/04/2018 09:35:09 PM---Folks- The other day, I posted an
> experimental patch [1] with an effectively
> 
> From: Eric Fried 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Date: 09/04/2018 09:35 PM
> Subject: [openstack-dev] [tempest][CI][nova compute] Skipping
> non-compute-driver tests
> 
> 
> 
> 
> 
> Folks-
> 
> The other day, I posted an experimental patch [1] with an effectively
> empty ComputeDriver (just enough to make n-cpu actually start) to see
> how much of our CI would pass. The theory being that any tests that
> still pass are tests that don't touch our compute driver, and are
> therefore not useful to run in our CI environment. Because anything that
> doesn't touch our code should already be well covered by generic
> dsvm-tempest CIs. The results [2] show that 707 tests still pass.
> 
> So I'm wondering whether there might be a way to mark tests as being
> "compute driver-specific" such that we could switch off all the other
> ones [3] via a one-line conf setting. Because surely this has potential
> to save a lot of CI resource not just for us but for other driver
> vendors, in tree and out.
> 
> Thanks,
> efried
> 
> [1] https://review.openstack.org/#/c/599066/
> [2]
> http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz
> [3] I get that there's still value in running all those tests. But it
> could be done like once every 10 or 50 or 100 runs instead of every time.
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Chen CH Ji]

I see the patch is still ongoing status and do you have a follow up
plan/discussion for that? we are maintaining 2 CIs (z/VM and KVM on z)
so skip non-compute related cases will be a good for 3rd part CI .. thanks

Best Regards!

Kevin (Chen) Ji 纪 晨

Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN   Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC



From:   Eric Fried 
To: "OpenStack Development Mailing List (not for usage questions)"

Date:   09/04/2018 09:35 PM
Subject:[openstack-dev] [tempest][CI][nova compute] Skipping
non-compute-driver tests



Folks-

 The other day, I posted an experimental patch [1] with an
effectively
empty ComputeDriver (just enough to make n-cpu actually start) to see
how much of our CI would pass. The theory being that any tests that
still pass are tests that don't touch our compute driver, and are
therefore not useful to run in our CI environment. Because anything that
doesn't touch our code should already be well covered by generic
dsvm-tempest CIs. The results [2] show that 707 tests still pass.

 So I'm wondering whether there might be a way to mark tests as
being
"compute driver-specific" such that we could switch off all the other
ones [3] via a one-line conf setting. Because surely this has potential
to save a lot of CI resource not just for us but for other driver
vendors, in tree and out.

Thanks,
efried

[1]
https://review.openstack.org/#/c/599066/

[2]
http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz

[3] I get that there's still value in running all those tests. But it
could be done like once every 10 or 50 or 100 runs instead of every time.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][CI][nova compute] Skipping non-compute-driver tests

[Eric Fried]

Folks-

The other day, I posted an experimental patch [1] with an effectively
empty ComputeDriver (just enough to make n-cpu actually start) to see
how much of our CI would pass. The theory being that any tests that
still pass are tests that don't touch our compute driver, and are
therefore not useful to run in our CI environment. Because anything that
doesn't touch our code should already be well covered by generic
dsvm-tempest CIs. The results [2] show that 707 tests still pass.

So I'm wondering whether there might be a way to mark tests as being
"compute driver-specific" such that we could switch off all the other
ones [3] via a one-line conf setting. Because surely this has potential
to save a lot of CI resource not just for us but for other driver
vendors, in tree and out.

Thanks,
efried

[1] https://review.openstack.org/#/c/599066/
[2]
http://184.172.12.213/66/599066/5/check/nova-powervm-out-of-tree-pvm/a1b42d5/powervm_os_ci.html.gz
[3] I get that there's still value in running all those tests. But it
could be done like once every 10 or 50 or 100 runs instead of every time.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] trouble setting tempest feature flag

[Ghanshyam Mann]

  On Wed, 29 Aug 2018 08:20:37 +0900 Eric K  
wrote  
 > Ha. Turned out to be a simple mistake in hyphens vs underscores.

Thanks for update and good to know it is resolved now. Sorry I could not 
checked this further due to PTO.

-gmann

 > On Tue, Aug 28, 2018 at 3:06 PM Eric K  wrote:
 > >
 > > Any thoughts on what could be going wrong that the tempest tests still
 > > see the default conf values rather than those set here? Thanks lots!
 > >
 > > Here is the devstack log line showing the flags being set:
 > > http://logs.openstack.org/64/594564/4/check/congress-devstack-api-mysql/ce34264/logs/devstacklog.txt.gz#_2018-08-28_21_23_15_934
 > >
 > > On Wed, Aug 22, 2018 at 9:12 AM Eric K  wrote:
 > > >
 > > > Hi all,
 > > >
 > > > I have added feature flags for the congress tempest plugin [1] and set
 > > > them in the devstack plugin [2], but the flags seem to be ignored. The
 > > > tests are skipped [3] according to the default False flag rather than
 > > > run according to the True flag set in devstack plugin. Any hints on
 > > > what may be wrong? Thanks so much!
 > > >
 > > > [1] https://review.openstack.org/#/c/594747/3
 > > > [2] https://review.openstack.org/#/c/594793/1/devstack/plugin.sh
 > > > [3] 
 > > > http://logs.openstack.org/64/594564/3/check/congress-devstack-api-mysql/b2cd46f/logs/testr_results.html.gz
 > > > (the bottom two skipped tests were expected to run)
 > 
 > __
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 > 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] trouble setting tempest feature flag

[Eric K]

Ha. Turned out to be a simple mistake in hyphens vs underscores.
On Tue, Aug 28, 2018 at 3:06 PM Eric K  wrote:
>
> Any thoughts on what could be going wrong that the tempest tests still
> see the default conf values rather than those set here? Thanks lots!
>
> Here is the devstack log line showing the flags being set:
> http://logs.openstack.org/64/594564/4/check/congress-devstack-api-mysql/ce34264/logs/devstacklog.txt.gz#_2018-08-28_21_23_15_934
>
> On Wed, Aug 22, 2018 at 9:12 AM Eric K  wrote:
> >
> > Hi all,
> >
> > I have added feature flags for the congress tempest plugin [1] and set
> > them in the devstack plugin [2], but the flags seem to be ignored. The
> > tests are skipped [3] according to the default False flag rather than
> > run according to the True flag set in devstack plugin. Any hints on
> > what may be wrong? Thanks so much!
> >
> > [1] https://review.openstack.org/#/c/594747/3
> > [2] https://review.openstack.org/#/c/594793/1/devstack/plugin.sh
> > [3] 
> > http://logs.openstack.org/64/594564/3/check/congress-devstack-api-mysql/b2cd46f/logs/testr_results.html.gz
> > (the bottom two skipped tests were expected to run)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] trouble setting tempest feature flag

[Eric K]

Any thoughts on what could be going wrong that the tempest tests still
see the default conf values rather than those set here? Thanks lots!

Here is the devstack log line showing the flags being set:
http://logs.openstack.org/64/594564/4/check/congress-devstack-api-mysql/ce34264/logs/devstacklog.txt.gz#_2018-08-28_21_23_15_934

On Wed, Aug 22, 2018 at 9:12 AM Eric K  wrote:
>
> Hi all,
>
> I have added feature flags for the congress tempest plugin [1] and set
> them in the devstack plugin [2], but the flags seem to be ignored. The
> tests are skipped [3] according to the default False flag rather than
> run according to the True flag set in devstack plugin. Any hints on
> what may be wrong? Thanks so much!
>
> [1] https://review.openstack.org/#/c/594747/3
> [2] https://review.openstack.org/#/c/594793/1/devstack/plugin.sh
> [3] 
> http://logs.openstack.org/64/594564/3/check/congress-devstack-api-mysql/b2cd46f/logs/testr_results.html.gz
> (the bottom two skipped tests were expected to run)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][qa][congress] trouble setting tempest feature flag

[Eric K]

Hi all,

I have added feature flags for the congress tempest plugin [1] and set
them in the devstack plugin [2], but the flags seem to be ignored. The
tests are skipped [3] according to the default False flag rather than
run according to the True flag set in devstack plugin. Any hints on
what may be wrong? Thanks so much!

[1] https://review.openstack.org/#/c/594747/3
[2] https://review.openstack.org/#/c/594793/1/devstack/plugin.sh
[3] 
http://logs.openstack.org/64/594564/3/check/congress-devstack-api-mysql/b2cd46f/logs/testr_results.html.gz
(the bottom two skipped tests were expected to run)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] help with tempest plugin jobs against stable/queens

[Eric K]

On Tue, Aug 14, 2018 at 9:34 PM, Ghanshyam Mann  wrote:
>   On Wed, 15 Aug 2018 09:37:18 +0900 Eric K  
> wrote 
>  > I'm adding jobs [1] to the tempest plugin to run tests against
>  > congress stable/queens. The job output seems to show stable/queens
>  > getting checked out [2], but I know the test is *not* run against
>  > queens because it's using features not available in queens. The
>  > expected result is for several tests to fail as seen here [3]. All
>  > hints and tips much appreciated!
>
> You are doing it in right way by 'override-checkout: stable/queens'. And as 
> log also show, congress is checkout from stable/queens. I tried to check the 
> results but could not get what tests should fail and why.
>
> If you can give me more idea, i can debug that.
>
> -gmann
Thanks so much gmann!
For example, looking at
'congress_tempest_plugin.tests.scenario.congress_datasources.test_vitrage.TestVitrageDriver'
here:
http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql/36bacbe/logs/testr_results.html.gz

It shows passing 1 of 1, but that feature is not in the queens branch
at all. The expected result can be seen here:
http://logs.openstack.org/05/591805/2/check/congress-devstack-api-mysql/7d7b28e/logs/testr_results.html.gz
>
>  >
>  > [1] https://review.openstack.org/#/c/591861/1
>  > [2] 
> http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql-queens/f7b5752/job-output.txt.gz#_2018-08-14_22_30_36_899501
>  > [3] https://review.openstack.org/#/c/591805/ (the depends-on is
>  > irrelevant because that patch has been merged)
>  >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] tempest test conditioning on release version

[Eric K]

On Tue, Aug 14, 2018 at 8:59 PM, Ghanshyam Mann  wrote:
>   On Wed, 15 Aug 2018 06:40:57 +0900 Eric K  
> wrote 
>  > Anyone have an example handy of a tempest test conditioning on service
>  > release version (because new features not available in past versions)?
>  > Seems like it could get pretty messy and haphazard, so I'm curious to
>  > see best practices. Thanks lots!
>
> Thanks Eric for query. We do it in many times in Tempest and similar approach 
> can be adopt by tempest plugins. There are 2 ways we can handle  this-
>
> 1. Using feature flag. Tempest documentation is here [1].
>  Step1- This is simply adding a config options(feature flag) for new/old 
> feature.
>  Example- https://review.openstack.org/#/c/545627/   
> https://github.com/openstack/tempest/blob/6a8d495192632fd18dce4baf1a4b213f401a0167/tempest/config.py#L242
>  Step2- Based on that flag you can skip the tests where that feature is not 
> available.
>  Example-  
> https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/identity/base.py#L315
>  Step3- For gate, devstack plugin on project side (congress is your case [2]) 
> which is branch aware can set that flag to true and false based on which 
> branch that test is running. For tempest we do the same from 
> devstack/lib/tempest
>  Example - https://review.openstack.org/#/c/545680/
> https://github.com/openstack-dev/devstack/blob/8c1052001629d62f001d04c182500fa293858f47/lib/tempest#L308
>  Step4- For cloud testing(non-gate), tester can manually configure the those 
> flag based on what service version they are testing.
>
> 2. Detecting service version via version API
> - If you can get the service version info from API then you can use that 
> while skipping the tests.
> - One example if for compute where based on microversion, it can be detected 
> that test running against which release.
> - Example- 
> https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/compute/base.py#L114
>
>
> [1] 
> https://docs.openstack.org/tempest/latest/HACKING.html#branchless-tempest-considerations
> [2] 
> https://github.com/openstack/congress/blob/014361c809517661264d0364eaf1e261e449ea80/devstack/plugin.sh#L88
>
>  >
>  > Eric Kao

Thank you so much, Ghanshyam!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] help with tempest plugin jobs against stable/queens

[Ghanshyam Mann]

  On Wed, 15 Aug 2018 09:37:18 +0900 Eric K  
wrote  
 > I'm adding jobs [1] to the tempest plugin to run tests against
 > congress stable/queens. The job output seems to show stable/queens
 > getting checked out [2], but I know the test is *not* run against
 > queens because it's using features not available in queens. The
 > expected result is for several tests to fail as seen here [3]. All
 > hints and tips much appreciated!

You are doing it in right way by 'override-checkout: stable/queens'. And as log 
also show, congress is checkout from stable/queens. I tried to check the 
results but could not get what tests should fail and why. 

If you can give me more idea, i can debug that. 

-gmann

 > 
 > [1] https://review.openstack.org/#/c/591861/1
 > [2] 
 > http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql-queens/f7b5752/job-output.txt.gz#_2018-08-14_22_30_36_899501
 > [3] https://review.openstack.org/#/c/591805/ (the depends-on is
 > irrelevant because that patch has been merged)
 > 
 > __
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 > 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][congress] tempest test conditioning on release version

[Ghanshyam Mann]

  On Wed, 15 Aug 2018 06:40:57 +0900 Eric K  
wrote  
 > Anyone have an example handy of a tempest test conditioning on service
 > release version (because new features not available in past versions)?
 > Seems like it could get pretty messy and haphazard, so I'm curious to
 > see best practices. Thanks lots!

Thanks Eric for query. We do it in many times in Tempest and similar approach 
can be adopt by tempest plugins. There are 2 ways we can handle  this-

1. Using feature flag. Tempest documentation is here [1].
 Step1- This is simply adding a config options(feature flag) for new/old 
feature. 
 Example- https://review.openstack.org/#/c/545627/   
https://github.com/openstack/tempest/blob/6a8d495192632fd18dce4baf1a4b213f401a0167/tempest/config.py#L242
 Step2- Based on that flag you can skip the tests where that feature is not 
available. 
 Example-  
https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/identity/base.py#L315
 Step3- For gate, devstack plugin on project side (congress is your case [2]) 
which is branch aware can set that flag to true and false based on which branch 
that test is running. For tempest we do the same from devstack/lib/tempest
 Example - https://review.openstack.org/#/c/545680/
https://github.com/openstack-dev/devstack/blob/8c1052001629d62f001d04c182500fa293858f47/lib/tempest#L308
 Step4- For cloud testing(non-gate), tester can manually configure the those 
flag based on what service version they are testing. 

2. Detecting service version via version API
- If you can get the service version info from API then you can use that while 
skipping the tests.
- One example if for compute where based on microversion, it can be detected 
that test running against which release. 
- Example- 
https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/compute/base.py#L114


[1] 
https://docs.openstack.org/tempest/latest/HACKING.html#branchless-tempest-considerations
[2] 
https://github.com/openstack/congress/blob/014361c809517661264d0364eaf1e261e449ea80/devstack/plugin.sh#L88

 > 
 > Eric Kao
 > 
 > __
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 > 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][qa][congress] help with tempest plugin jobs against stable/queens

[Eric K]

I'm adding jobs [1] to the tempest plugin to run tests against
congress stable/queens. The job output seems to show stable/queens
getting checked out [2], but I know the test is *not* run against
queens because it's using features not available in queens. The
expected result is for several tests to fail as seen here [3]. All
hints and tips much appreciated!

[1] https://review.openstack.org/#/c/591861/1
[2] 
http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql-queens/f7b5752/job-output.txt.gz#_2018-08-14_22_30_36_899501
[3] https://review.openstack.org/#/c/591805/ (the depends-on is
irrelevant because that patch has been merged)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][qa][congress] tempest test conditioning on release version

[Eric K]

Anyone have an example handy of a tempest test conditioning on service
release version (because new features not available in past versions)?
Seems like it could get pretty messy and haphazard, so I'm curious to
see best practices. Thanks lots!

Eric Kao

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Small doubt in Tempest setup

[Goutham Pratapa]

stestr worked thanks but im getting the same error for

ostestr -l

any idea on what to do ??


On Mon, Aug 6, 2018 at 7:27 PM, Attila Fazekas  wrote:

> I was tried to be quick and become wrong. ;-)
>
> Here are the working ways:
>
> On Mon, Aug 6, 2018 at 3:49 PM, Attila Fazekas 
> wrote:
>
>> Please use ostestr or stestr instead of testr.
>>
>> $ git clone https://github.com/openstack/tempest
>> $ cd tempest/
>> $ stestr init
>> $ stestr list
>>
>> $ git clone https://github.com/openstack/tempest
>> $ cd tempest/
>> $ ostestr -l #old way, also worked, does to steps
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Cheers !!!
Goutham Pratapa
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Small doubt in Tempest setup

[Goutham Pratapa]

Done thanks afazekas

Thanks
Goutham

On Mon, 6 Aug 2018 at 7:27 PM, Attila Fazekas  wrote:

> I was tried to be quick and become wrong. ;-)
>
> Here are the working ways:
>
> On Mon, Aug 6, 2018 at 3:49 PM, Attila Fazekas 
> wrote:
>
>> Please use ostestr or stestr instead of testr.
>>
>> $ git clone https://github.com/openstack/tempest
>> $ cd tempest/
>> $ stestr init
>> $ stestr list
>>
>> $ git clone https://github.com/openstack/tempest
>> $ cd tempest/
>> $ ostestr -l #old way, also worked, does to steps
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
Cheers !!!
Goutham Pratapa
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Small doubt in Tempest setup

[Attila Fazekas]

I was tried to be quick and become wrong. ;-)

Here are the working ways:

On Mon, Aug 6, 2018 at 3:49 PM, Attila Fazekas  wrote:

> Please use ostestr or stestr instead of testr.
>
> $ git clone https://github.com/openstack/tempest
> $ cd tempest/
> $ stestr init
> $ stestr list
>
> $ git clone https://github.com/openstack/tempest
> $ cd tempest/
> $ ostestr -l #old way, also worked, does to steps
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Small doubt in Tempest setup

[Attila Fazekas]

Please use ostestr or stestr instead of testr.

$ git clone https://github.com/openstack/tempest
$ cd tempest/
$ stestr --list

$ ostestr -l #old way, also worked

These tools handling the config creation implicitly.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Small doubt in Tempest setup

[Goutham Pratapa]

Hi all,

This is regarding Tempest setup I have cloned and setup my tempest i could
run my tests with

'*nosetests*' also but when i try  to run with *testr* im getting


*$ testr list-tests *


*No .testr.conf config file*

any idea why it is occurring and any idea how to fix it will really help..

Thanks in advance.

-- 
Cheers !!!
Goutham Pratapa
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Felipe Monteiro for Tempest core

[Ghanshyam Mann]

Thanks all for voting.

We have all voting in favor of Felipe and enough to add him as Core. I
will add him in Core list on gerrit.

Welcome to the team Felipe!!

-gmann



On Sat, Apr 28, 2018 at 7:27 PM, Ghanshyam Mann  wrote:
> Hi Tempest Team,
>
> I would like to propose  Felipe Monteiro (irc: felipemonteiro) to Tempest 
> core.
>
> Felipe has been an active contributor to the Tempest  since the Pike
> cycle. He has been doing lot of review and commits since then. Filling
> the gaps on service clients side and their testing and lot other
> areas. He has demonstrated the good quality and feedback while his
> review.
>
> He has good understanding of Tempest source code and project missions
> & goal. IMO his efforts are highly valuable and it will be great to
> have him in team.
>
>
> As per usual practice, please vote +1 or -1 to the nomination. I will
> keep this nomination open for a week or until everyone voted.
>
> Felipe Reviews and Commit -
> https://review.openstack.org/#/q/reviewer:felipe.monte...@att.com+project:openstack/tempest
> https://review.openstack.org/#/q/owner:felipe.monte...@att.com+project:openstack/tempest
>
> -gmann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Felipe Monteiro for Tempest core

[Andrea Frittoli]

+1!!



On Sat, 28 Apr 2018, 11:29 am Ghanshyam Mann, 
wrote:

> Hi Tempest Team,
>
> I would like to propose  Felipe Monteiro (irc: felipemonteiro) to Tempest
> core.
>
> Felipe has been an active contributor to the Tempest  since the Pike
> cycle. He has been doing lot of review and commits since then. Filling
> the gaps on service clients side and their testing and lot other
> areas. He has demonstrated the good quality and feedback while his
> review.
>
> He has good understanding of Tempest source code and project missions
> & goal. IMO his efforts are highly valuable and it will be great to
> have him in team.
>
>
> As per usual practice, please vote +1 or -1 to the nomination. I will
> keep this nomination open for a week or until everyone voted.
>
> Felipe Reviews and Commit -
>
> https://review.openstack.org/#/q/reviewer:felipe.monte...@att.com+project:openstack/tempest
>
> https://review.openstack.org/#/q/owner:felipe.monte...@att.com+project:openstack/tempest
>
> -gmann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Felipe Monteiro for Tempest core

[Ken'ichi Ohmichi]

+1
Thanks for your contribution, Felipe

2018年4月28日(土) 3:29 Ghanshyam Mann :

> Hi Tempest Team,
>
> I would like to propose  Felipe Monteiro (irc: felipemonteiro) to Tempest
> core.
>
> Felipe has been an active contributor to the Tempest  since the Pike
> cycle. He has been doing lot of review and commits since then. Filling
> the gaps on service clients side and their testing and lot other
> areas. He has demonstrated the good quality and feedback while his
> review.
>
> He has good understanding of Tempest source code and project missions
> & goal. IMO his efforts are highly valuable and it will be great to
> have him in team.
>
>
> As per usual practice, please vote +1 or -1 to the nomination. I will
> keep this nomination open for a week or until everyone voted.
>
> Felipe Reviews and Commit -
>
> https://review.openstack.org/#/q/reviewer:felipe.monte...@att.com+project:openstack/tempest
>
> https://review.openstack.org/#/q/owner:felipe.monte...@att.com+project:openstack/tempest
>
> -gmann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Felipe Monteiro for Tempest core

[Masayuki Igawa]

+1

-- Masayuki Igawa
  Key fingerprint = C27C 2F00 3A2A 999A 903A  753D 290F 53ED C899 BF89

On Sat, Apr 28, 2018, at 19:27, Ghanshyam Mann wrote:
> Hi Tempest Team,
> 
> I would like to propose  Felipe Monteiro (irc: felipemonteiro) to Tempest 
> core.
> 
> Felipe has been an active contributor to the Tempest  since the Pike
> cycle. He has been doing lot of review and commits since then. Filling
> the gaps on service clients side and their testing and lot other
> areas. He has demonstrated the good quality and feedback while his
> review.
> 
> He has good understanding of Tempest source code and project missions
> & goal. IMO his efforts are highly valuable and it will be great to
> have him in team.
> 
> 
> As per usual practice, please vote +1 or -1 to the nomination. I will
> keep this nomination open for a week or until everyone voted.
> 
> Felipe Reviews and Commit -
> https://review.openstack.org/#/q/reviewer:felipe.monte...@att.com+project:openstack/tempest
> https://review.openstack.org/#/q/owner:felipe.monte...@att.com+project:openstack/tempest
> 
> -gmann
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Proposing Felipe Monteiro for Tempest core

[Ghanshyam Mann]

Hi Tempest Team,

I would like to propose  Felipe Monteiro (irc: felipemonteiro) to Tempest core.

Felipe has been an active contributor to the Tempest  since the Pike
cycle. He has been doing lot of review and commits since then. Filling
the gaps on service clients side and their testing and lot other
areas. He has demonstrated the good quality and feedback while his
review.

He has good understanding of Tempest source code and project missions
& goal. IMO his efforts are highly valuable and it will be great to
have him in team.


As per usual practice, please vote +1 or -1 to the nomination. I will
keep this nomination open for a week or until everyone voted.

Felipe Reviews and Commit -
https://review.openstack.org/#/q/reviewer:felipe.monte...@att.com+project:openstack/tempest
https://review.openstack.org/#/q/owner:felipe.monte...@att.com+project:openstack/tempest

-gmann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Tempest plugin for Neutron Stadium projects

[Bernard Cafarelli]

That makes sense, thanks! I will add the topic to the next
networking-sfc meeting

On 2 January 2018 at 17:06, Miguel Lavalle  wrote:
> Hi Neutron community,
>
> During the last Neutron drivers meeting, we discussed whether all the
> Neutron Stadium projects should have their Tempest code in
> https://github.com/openstack/neutron-tempest-plugin/. It was decided to
> allow Stadium projects to get their tests in the consolidated plugin but it
> will not be a requirement. The assumption is that many projects might be
> stretched in resources and we don't want to create more work for them.
>
> Best regards
>
> Miguel
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Bernard Cafarelli

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Tempest plugin for Neutron Stadium projects

[Miguel Lavalle]

Hi Neutron community,

During the last Neutron drivers meeting, we discussed whether all the
Neutron Stadium projects should have their Tempest code in
https://github.com/openstack/neutron-tempest-plugin/. It was decided to
allow Stadium projects to get their tests in the consolidated plugin but it
will not be a requirement. The assumption is that many projects might be
stretched in resources and we don't want to create more work for them.

Best regards

Miguel
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][infra][congress] subprocess launched in tempest test lacks file permission

[Eric K]

Thanks for the suggestion, Jeremy.

I¹ll look into that. Two key pieces will be needed:
1. Know ahead of time what user tempest tests would be run from (should be
determined by job setup?)
2. Have a way to specify in job/devstack to start congress using that user.

On 7/28/17, 6:30 AM, "Jeremy Stanley"  wrote:

>On 2017-07-27 20:37:49 -0700 (-0700), Eric K wrote:
>> A tempest test [1] launches additional instances of Congress using
>> subprocess.popen and tests the coordination between them and the
>>original
>> instance launched by devstack. The problem is, the new instances are
>> launched from the tempest test user rather than the user of the original
>> congress instance devstack created. As a result, the new instances fail
>> for being unable to access the encryption keys created by the original
>> congress instance (600 permission).[2]
>> 
>> Any suggestions for how to workaround this problem? Is it possible to
>> somehow configure the gate job to run tempest tests as SU or as the
>> stackuser that launches the original congress instance?
>[...]
>
>Could you flip this around and create the original instances with
>the tempest user? That seems less likely to violate base assumptions
>about the test environment.
>-- 
>Jeremy Stanley
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][infra][congress] subprocess launched in tempest test lacks file permission

[Eric K]

Thanks for your help and sketch of solution, Andrea!  -Eric

From:  Andrea Frittoli 
Reply-To:  "OpenStack Development Mailing List (not for usage questions)"

Date:  Friday, July 28, 2017 at 9:03 AM
To:  "OpenStack Development Mailing List (not for usage questions)"

Subject:  Re: [openstack-dev] [tempest][infra][congress] subprocess launched
in tempest test lacks file permission

> 
> 
> On Fri, Jul 28, 2017 at 4:38 AM Eric K  wrote:
>> A tempest test [1] launches additional instances of Congress using
>> subprocess.popen and tests the coordination between them and the original
>> instance launched by devstack. The problem is, the new instances are
>> launched from the tempest test user rather than the user of the original
>> congress instance devstack created. As a result, the new instances fail
>> for being unable to access the encryption keys created by the original
>> congress instance (600 permission).[2]
>> 
>> Any suggestions for how to workaround this problem? Is it possible to
>> somehow configure the gate job to run tempest tests as SU or as the
>> stackuser that launches the original congress instance? Thanks so much!
> 
> Unfortunately we don't have any facility in Tempest to control the user for
> subprocesses since Tempest is meant for API testing only. Starting a
> subprocess within the test sounds a bit outside of the scope of what you
> may want to do in a Tempest test.
>  
> If you want to use Tempest to verify HA type of scenarios you may want
> to move the orchestration bit outside of Tempest - you could have an ansible
> script that manages your services and run Tempest before and after certain
> actions to verify that things are still working.
> 
> If you want tests to run in parallel to things happening to your system you
> can do that but again you will need an external component to orchestrate
> this, and you don't know what kind of tempest test will be running unless
> you use a single long running scenario test designed for this purpose.
> 
> Andrea Frittoli (andreaf)
> 
>> 
>> [1]
>> https://github.com/openstack/congress/blob/master/congress_tempest_tests/te
>> sts/scenario/congress_ha/test_ha.py.disabled#L117
>> <https://github.com/openstack/congress/blob/master/congress_tempest_tests/tes
>> ts/scenario/congress_ha/test_ha.py.disabled#L117>  (currently disabled,
>> trying to re-enable)
>> 
>> [2]
>> http://logs.openstack.org/35/487235/5/check/gate-congress-dsvm-api-mysql-ub
>> untu-xenial/607623d/logs/testr_results.html.gz
>> <http://logs.openstack.org/35/487235/5/check/gate-congress-dsvm-api-mysql-ubu
>> ntu-xenial/607623d/logs/testr_results.html.gz>  (find: OSError: [Errno 13]
>> Permission denied: '/etc/congress/keys/aes_key¹)
>> 
>> 
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions) Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][infra][congress] subprocess launched in tempest test lacks file permission

[Andrea Frittoli]

On Fri, Jul 28, 2017 at 4:38 AM Eric K  wrote:

> A tempest test [1] launches additional instances of Congress using
> subprocess.popen and tests the coordination between them and the original
> instance launched by devstack. The problem is, the new instances are
> launched from the tempest test user rather than the user of the original
> congress instance devstack created. As a result, the new instances fail
> for being unable to access the encryption keys created by the original
> congress instance (600 permission).[2]
>
> Any suggestions for how to workaround this problem? Is it possible to
> somehow configure the gate job to run tempest tests as SU or as the
> stackuser that launches the original congress instance? Thanks so much!
>

Unfortunately we don't have any facility in Tempest to control the user for
subprocesses since Tempest is meant for API testing only. Starting a
subprocess within the test sounds a bit outside of the scope of what you
may want to do in a Tempest test.

If you want to use Tempest to verify HA type of scenarios you may want
to move the orchestration bit outside of Tempest - you could have an ansible
script that manages your services and run Tempest before and after certain
actions to verify that things are still working.

If you want tests to run in parallel to things happening to your system you
can do that but again you will need an external component to orchestrate
this, and you don't know what kind of tempest test will be running unless
you use a single long running scenario test designed for this purpose.

Andrea Frittoli (andreaf)


> [1]
> https://github.com/openstack/congress/blob/master/congress_tempest_tests/te
> sts/scenario/congress_ha/test_ha.py.disabled#L117
> 
> (currently disabled,
> trying to re-enable)
>
> [2]
> http://logs.openstack.org/35/487235/5/check/gate-congress-dsvm-api-mysql-ub
> untu-xenial/607623d/logs/testr_results.html.gz
> 
> (find: OSError: [Errno 13]
> Permission denied: '/etc/congress/keys/aes_key¹)
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][infra][congress] subprocess launched in tempest test lacks file permission

[Jeremy Stanley]

On 2017-07-27 20:37:49 -0700 (-0700), Eric K wrote:
> A tempest test [1] launches additional instances of Congress using
> subprocess.popen and tests the coordination between them and the original
> instance launched by devstack. The problem is, the new instances are
> launched from the tempest test user rather than the user of the original
> congress instance devstack created. As a result, the new instances fail
> for being unable to access the encryption keys created by the original
> congress instance (600 permission).[2]
> 
> Any suggestions for how to workaround this problem? Is it possible to
> somehow configure the gate job to run tempest tests as SU or as the
> stackuser that launches the original congress instance?
[...]

Could you flip this around and create the original instances with
the tempest user? That seems less likely to violate base assumptions
about the test environment.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][infra][congress] subprocess launched in tempest test lacks file permission

[Eric K]

A tempest test [1] launches additional instances of Congress using
subprocess.popen and tests the coordination between them and the original
instance launched by devstack. The problem is, the new instances are
launched from the tempest test user rather than the user of the original
congress instance devstack created. As a result, the new instances fail
for being unable to access the encryption keys created by the original
congress instance (600 permission).[2]

Any suggestions for how to workaround this problem? Is it possible to
somehow configure the gate job to run tempest tests as SU or as the
stackuser that launches the original congress instance? Thanks so much!

[1] 
https://github.com/openstack/congress/blob/master/congress_tempest_tests/te
sts/scenario/congress_ha/test_ha.py.disabled#L117 (currently disabled,
trying to re-enable)

[2] 
http://logs.openstack.org/35/487235/5/check/gate-congress-dsvm-api-mysql-ub
untu-xenial/607623d/logs/testr_results.html.gz (find: OSError: [Errno 13]
Permission denied: '/etc/congress/keys/aes_key¹)



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Andrea Frittoli]

Polls are closed now, with six positive votes.
Welcome to the team Fanglei!

andrea

On Thu, May 18, 2017 at 10:59 AM Attila Fazekas  wrote:

> +1, Totally agree.
>
> Best Regards,
> Attila
>
> On Tue, May 16, 2017 at 10:22 AM, Andrea Frittoli <
> andrea.fritt...@gmail.com> wrote:
>
>> Hello team,
>>
>> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
>>
>> Over the past two cycle Fanglei has been steadily contributing to Tempest
>> and its community.
>> She's done a great deal of work in making Tempest code cleaner, easier to
>> read, maintain and
>> debug, fixing bugs and removing cruft. Both her code as well as her
>> reviews demonstrate a
>> very good understanding of Tempest internals and of the project future
>> direction.
>> I believe Fanglei will make an excellent addition to the team.
>>
>> As per the usual, if the current Tempest core team members would please
>> vote +1
>> or -1(veto) to the nomination when you get a chance. We'll keep the polls
>> open
>> for 5 days or until everyone has voted.
>>
>> References:
>> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
>> https://review.openstack.org/#/q/reviewer:zhufl
>>
>> Thank you,
>>
>> Andrea (andreaf)
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Attila Fazekas]

+1, Totally agree.

Best Regards,
Attila

On Tue, May 16, 2017 at 10:22 AM, Andrea Frittoli  wrote:

> Hello team,
>
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
>
> Over the past two cycle Fanglei has been steadily contributing to Tempest
> and its community.
> She's done a great deal of work in making Tempest code cleaner, easier to
> read, maintain and
> debug, fixing bugs and removing cruft. Both her code as well as her
> reviews demonstrate a
> very good understanding of Tempest internals and of the project future
> direction.
> I believe Fanglei will make an excellent addition to the team.
>
> As per the usual, if the current Tempest core team members would please
> vote +1
> or -1(veto) to the nomination when you get a chance. We'll keep the polls
> open
> for 5 days or until everyone has voted.
>
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
> https://review.openstack.org/#/q/reviewer:zhufl
>
> Thank you,
>
> Andrea (andreaf)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Ken'ichi Ohmichi]

+1, thanks Zhu for your work.

2017-05-16 4:22 GMT-04:00 Andrea Frittoli :
> Hello team,
>
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
>
> Over the past two cycle Fanglei has been steadily contributing to Tempest
> and its community.
> She's done a great deal of work in making Tempest code cleaner, easier to
> read, maintain and
> debug, fixing bugs and removing cruft. Both her code as well as her reviews
> demonstrate a
> very good understanding of Tempest internals and of the project future
> direction.
> I believe Fanglei will make an excellent addition to the team.
>
> As per the usual, if the current Tempest core team members would please vote
> +1
> or -1(veto) to the nomination when you get a chance. We'll keep the polls
> open
> for 5 days or until everyone has voted.
>
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
> https://review.openstack.org/#/q/reviewer:zhufl
>
> Thank you,
>
> Andrea (andreaf)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Masayuki Igawa]

+1!

-- 
  Masayuki Igawa
  masay...@igawa.me



On Tue, May 16, 2017, at 05:22 PM, Andrea Frittoli wrote:
> Hello team,
> 
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
> 
> Over the past two cycle Fanglei has been steadily contributing to
> Tempest and its community.> She's done a great deal of work in making Tempest 
> code cleaner, easier
> to read, maintain and> debug, fixing bugs and removing cruft. Both her code 
> as well as her
> reviews demonstrate a> very good understanding of Tempest internals and of 
> the project future
> direction.> I believe Fanglei will make an excellent addition to the team.
> 
> As per the usual, if the current Tempest core team members would
> please vote +1> or -1(veto) to the nomination when you get a chance. We'll 
> keep the
> polls open> for 5 days or until everyone has voted.
> 
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn 
> https://review.openstack.org/#/q/reviewer:zhufl 
> 
> Thank you,
> 
> Andrea (andreaf)
> -
> > OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Masayuki Igawa]

+1!

-- 
  Masayuki Igawa
  masay...@igawa.me



On Tue, May 16, 2017, at 05:22 PM, Andrea Frittoli wrote:
> Hello team,
> 
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
> 
> Over the past two cycle Fanglei has been steadily contributing to
> Tempest and its community.> She's done a great deal of work in making Tempest 
> code cleaner, easier
> to read, maintain and> debug, fixing bugs and removing cruft. Both her code 
> as well as her
> reviews demonstrate a> very good understanding of Tempest internals and of 
> the project future
> direction.> I believe Fanglei will make an excellent addition to the team.
> 
> As per the usual, if the current Tempest core team members would
> please vote +1> or -1(veto) to the nomination when you get a chance. We'll 
> keep the
> polls open> for 5 days or until everyone has voted.
> 
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn 
> https://review.openstack.org/#/q/reviewer:zhufl 
> 
> Thank you,
> 
> Andrea (andreaf)
> -
> > OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Matthew Treinish]

On Tue, May 16, 2017 at 08:22:44AM +, Andrea Frittoli wrote:
> Hello team,
> 
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
> 
> Over the past two cycle Fanglei has been steadily contributing to Tempest
> and its community.
> She's done a great deal of work in making Tempest code cleaner, easier to
> read, maintain and
> debug, fixing bugs and removing cruft. Both her code as well as her reviews
> demonstrate a
> very good understanding of Tempest internals and of the project future
> direction.
> I believe Fanglei will make an excellent addition to the team.
> 
> As per the usual, if the current Tempest core team members would please
> vote +1
> or -1(veto) to the nomination when you get a chance. We'll keep the polls
> open
> for 5 days or until everyone has voted.

+1

-Matt Treinish

> 
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
> https://review.openstack.org/#/q/reviewer:zhufl


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Ghanshyam Mann]

+1. Nice work done by Fanglei and good to have her in team.

-gmann


On Tue, May 16, 2017 at 5:22 PM, Andrea Frittoli
 wrote:
> Hello team,
>
> I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.
>
> Over the past two cycle Fanglei has been steadily contributing to Tempest
> and its community.
> She's done a great deal of work in making Tempest code cleaner, easier to
> read, maintain and
> debug, fixing bugs and removing cruft. Both her code as well as her reviews
> demonstrate a
> very good understanding of Tempest internals and of the project future
> direction.
> I believe Fanglei will make an excellent addition to the team.
>
> As per the usual, if the current Tempest core team members would please vote
> +1
> or -1(veto) to the nomination when you get a chance. We'll keep the polls
> open
> for 5 days or until everyone has voted.
>
> References:
> https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
> https://review.openstack.org/#/q/reviewer:zhufl
>
> Thank you,
>
> Andrea (andreaf)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Proposing Fanglei Zhu for Tempest core

[Andrea Frittoli]

Hello team,

I'm very pleased to propose Fanglei Zhu (zhufl) for Tempest core.

Over the past two cycle Fanglei has been steadily contributing to Tempest
and its community.
She's done a great deal of work in making Tempest code cleaner, easier to
read, maintain and
debug, fixing bugs and removing cruft. Both her code as well as her reviews
demonstrate a
very good understanding of Tempest internals and of the project future
direction.
I believe Fanglei will make an excellent addition to the team.

As per the usual, if the current Tempest core team members would please
vote +1
or -1(veto) to the nomination when you get a chance. We'll keep the polls
open
for 5 days or until everyone has voted.

References:
https://review.openstack.org/#/q/owner:zhu.fanglei%2540zte.com.cn
https://review.openstack.org/#/q/reviewer:zhufl

Thank you,

Andrea (andreaf)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Jenkins verification failures with stable release gates

[Andrea Frittoli]

On Mon, May 15, 2017 at 2:51 PM Hemanth N  wrote:

> Hi
>
> I have written new Tempest functional test cases for Identity OAUTH API.
> https://review.openstack.org/#/c/463240/
>
>
> This patch have dependency on a keystone fix that is still under
> review and I have mentioned this patch as Depends-On for above one.
> (https://review.openstack.org/#/c/464577/)
>
> The gates with stable releases are failing but on master it is successful
> gate-tempest-dsvm-neutron-full-ubuntu-xenial-ocata
> gate-tempest-dsvm-neutron-full-ubuntu-xenial-newton
>
> I am assuming the stable releases will cherry-pick the Depends-On
> patches and then builds/verifies the environment.
> Is my understanding correct?
>

Your change is for keystone master, and it won't be automatically
cherry-picked.


> If not, how should I proceed in such scenarios.
>

Your change make it possible to use the oath API in deployments where TLS
is terminated in front of the API server. I would use a feature flag on
Tempest side
to indicate whether this keystone feature is available. Default to false,
 set to true in Pike
in devstack, skip the tests if the feature is not available.

Even if your change was back ported, merging this would immediately enforce
the new behaviour on all Netwton+ clouds, to I think having a feature flag
would be
useful anyways.

andrea


> Thanks in Advance.
>
> Best Regards,
> Hemanth
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Jenkins verification failures with stable release gates

[Hemanth N]

Hi

I have written new Tempest functional test cases for Identity OAUTH API.
https://review.openstack.org/#/c/463240/


This patch have dependency on a keystone fix that is still under
review and I have mentioned this patch as Depends-On for above one.
(https://review.openstack.org/#/c/464577/)

The gates with stable releases are failing but on master it is successful
gate-tempest-dsvm-neutron-full-ubuntu-xenial-ocata
gate-tempest-dsvm-neutron-full-ubuntu-xenial-newton

I am assuming the stable releases will cherry-pick the Depends-On
patches and then builds/verifies the environment.
Is my understanding correct?
If not, how should I proceed in such scenarios.

Thanks in Advance.

Best Regards,
Hemanth

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest] Failing to run a test on a remote devstack server

[Jordan Pittier]

Hi Yossi,
I suggest you carefully read the source code of this test and all the
options in your tempest.conf file.

Jordan

On Thu, Oct 20, 2016 at 3:22 PM, Yossi Tamarov 
wrote:

> Hello everyone,
> When trying to run the next test: TestNetworkBasicOps.test_
> connectivity_between_vms_on_different_networks, from the devstack server
> itself, it passes, but when trying to run it from another server. i.e. by
> changing the *uri *parameter in the tempest.conf file in another server,
> the first step fails (
>
> I really need to run the test from a remote server.
> Does anyone have any idea of what the origin of the problem might be?
>
> thanks in advance,
> Joseph.
>
> Log prints:
>
> *The unsuccessful run's output:*
> *root@devstack-man17-d1:/opt/stack/tempest# ostestr --regex
> '(?!.*\[.*\bslow\b.*\])('tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_ne*
> *tworks')'*
> *running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \*
> *OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \*
> *OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \*
> *OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \*
> *${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./}
> ${OS_TEST_PATH:-./tempest/test_discover} --list*
> *running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \*
> *OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \*
> *OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \*
> *OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \*
> *${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./}
> ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpflGqv0*
> *{0}
> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks
> [1.439351s] ... FAILED*
>
> *==*
> *Failed 1 tests - output below:*
> *==*
>
>
> *tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks[compute,id-1546850e-fbaa-42f5-8b5f-03d8a6a95f15,network]*
>
> **
>
> *Captured pythonlogging:*
> *~~~*
> *2016-10-20 16:18:42,698 12084 INFO
> [tempest.lib.common.rest_client] Request
> (TestNetworkBasicOps:test_connectivity_between_vms_on_different_networks):
> 200 POST http://10.0.43.14:500 *
> *0/v2.0/tokens*
> *2016-10-20 16:18:42,698 12084 DEBUG
>  [tempest.lib.common.rest_client] Request - Headers: {'Content-Type':
> 'application/json', 'Accept': 'application/json'}*
> *Body: *
> *Response - Headers: {'status': '200', 'content-length': '3033',
> 'content-location': 'http://10.0.43.14:5000/v2.0/tokens
> ', 'vary': 'X-Auth-Token', 'server':
> 'Apache/2.4.10 (Debian)',*
> * 'connection': 'close', 'date': 'Thu, 20 Oct 2016 13:18:39 GMT',
> 'content-type': 'application/json', 'x-openstack-request-id':
> 'req-860cca80-c240-4868-af33-6767c34bac7b'}*
> *Body: {"access": {"token": {"issued_at":
> "2016-10-20T13:18:40.00Z", "expires": "2016-10-20T14:18:40Z", "id":
> "bcc6ea6da5a54551b8d48b2726fbc4cf", "tenant": {"description": "t*
> *empest-TestNetworkBasicOps-2126651196 <2126651196>-desc", "enabled":
> true, "id": "f8cb09dd19094615befd6cc27553c5d6", "name":
> "tempest-TestNetworkBasicOps-2126651196 <2126651196>"}, "audit_ids":
> ["5nc6OIUNQMiN3HtwZ7x*
> *i2A"]}, "serviceCatalog": [{"endpoints": [{"adminURL":
> "http://10.0.43.14:8774/v2.1/f8cb09dd19094615befd6cc27553c5d6
> ", "region":
> "RegionOne", "internalURL": "http://10.0.43.14:8774/v2.1/f8c
> *
> *b09dd19094615befd6cc27553c5d6", "id": "1f706e5df45a435c86e04536f1337d7a",
> "publicURL": "http://10.0.43.14:8774/v2.1/f8cb09dd19094615befd6cc27553c5d6
> "}],
> "endpoints_links": [], "type": "comp*
> *ute", "name": "nova"}, {"endpoints": [{"adminURL":
> "http://10.0.43.14:9696/ ", "region": "RegionOne",
> "internalURL": "http://10.0.43.14:9696/ ", "id":
> "32476f6907cc48d49c5bc5fe30bacd2b", "pub*
> *licURL": "http://10.0.43.14:9696/ "}],
> "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints":
> [{"adminURL": "http://10.0.43.14:8776/v2/f8cb09dd19094615befd6cc27553c5d6
> ",*
> * "region": "RegionOne", "internalURL":
> "http://10.0.43.14:8776/v2/f8cb09dd19094615befd6cc27553c5d6
> ", "id":
> "06fec1e446244e2d84c2fdced4fc51c7", "publicURL":
> "http://10.0.43.14:8776/v2/f8cb09 *
> *dd19094615befd6cc27553c5d6"}], "endpoints_

[openstack-dev] [Tempest] Failing to run a test on a remote devstack server

[Yossi Tamarov]

Hello everyone,
When trying to run the next test:
TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks, from
the devstack server itself, it passes, but when trying to run it from
another server. i.e. by changing the *uri *parameter in the tempest.conf
file in another server, the first step fails (

I really need to run the test from a remote server.
Does anyone have any idea of what the origin of the problem might be?

thanks in advance,
Joseph.

Log prints:

*The unsuccessful run's output:*
*root@devstack-man17-d1:/opt/stack/tempest# ostestr --regex
'(?!.*\[.*\bslow\b.*\])('tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_ne*
*tworks')'*
*running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \*
*OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \*
*OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \*
*OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \*
*${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./}
${OS_TEST_PATH:-./tempest/test_discover} --list*
*running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \*
*OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \*
*OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \*
*OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \*
*${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./}
${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpflGqv0*
*{0}
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks
[1.439351s] ... FAILED*

*==*
*Failed 1 tests - output below:*
*==*

*tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks[compute,id-1546850e-fbaa-42f5-8b5f-03d8a6a95f15,network]*
**

*Captured pythonlogging:*
*~~~*
*2016-10-20 16:18:42,698 12084 INFO
[tempest.lib.common.rest_client] Request
(TestNetworkBasicOps:test_connectivity_between_vms_on_different_networks):
200 POST http://10.0.43.14:500 *
*0/v2.0/tokens*
*2016-10-20 16:18:42,698 12084 DEBUG
 [tempest.lib.common.rest_client] Request - Headers: {'Content-Type':
'application/json', 'Accept': 'application/json'}*
*Body: *
*Response - Headers: {'status': '200', 'content-length': '3033',
'content-location': 'http://10.0.43.14:5000/v2.0/tokens
', 'vary': 'X-Auth-Token', 'server':
'Apache/2.4.10 (Debian)',*
* 'connection': 'close', 'date': 'Thu, 20 Oct 2016 13:18:39 GMT',
'content-type': 'application/json', 'x-openstack-request-id':
'req-860cca80-c240-4868-af33-6767c34bac7b'}*
*Body: {"access": {"token": {"issued_at":
"2016-10-20T13:18:40.00Z", "expires": "2016-10-20T14:18:40Z", "id":
"bcc6ea6da5a54551b8d48b2726fbc4cf", "tenant": {"description": "t*
*empest-TestNetworkBasicOps-2126651196-desc", "enabled": true, "id":
"f8cb09dd19094615befd6cc27553c5d6", "name":
"tempest-TestNetworkBasicOps-2126651196"}, "audit_ids":
["5nc6OIUNQMiN3HtwZ7x*
*i2A"]}, "serviceCatalog": [{"endpoints": [{"adminURL":
"http://10.0.43.14:8774/v2.1/f8cb09dd19094615befd6cc27553c5d6
", "region":
"RegionOne", "internalURL": "http://10.0.43.14:8774/v2.1/f8c
*
*b09dd19094615befd6cc27553c5d6", "id": "1f706e5df45a435c86e04536f1337d7a",
"publicURL": "http://10.0.43.14:8774/v2.1/f8cb09dd19094615befd6cc27553c5d6
"}],
"endpoints_links": [], "type": "comp*
*ute", "name": "nova"}, {"endpoints": [{"adminURL":
"http://10.0.43.14:9696/ ", "region": "RegionOne",
"internalURL": "http://10.0.43.14:9696/ ", "id":
"32476f6907cc48d49c5bc5fe30bacd2b", "pub*
*licURL": "http://10.0.43.14:9696/ "}],
"endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints":
[{"adminURL": "http://10.0.43.14:8776/v2/f8cb09dd19094615befd6cc27553c5d6
",*
* "region": "RegionOne", "internalURL":
"http://10.0.43.14:8776/v2/f8cb09dd19094615befd6cc27553c5d6
", "id":
"06fec1e446244e2d84c2fdced4fc51c7", "publicURL":
"http://10.0.43.14:8776/v2/f8cb09 *
*dd19094615befd6cc27553c5d6"}], "endpoints_links": [], "type": "volumev2",
"name": "cinderv2"}, {"endpoints": [{"adminURL": "http://10.0.43.14:9292
", "region": "RegionOne", "internalURL": "h*
*ttp://10.0.43.14:9292 ", "id":
"4dab7644ae384937973b5d460da0b8f7", "publicURL": "http://10.0.43.14:9292
"}], "endpoints_links": [], "type": "image",
"name": "gla

Re: [openstack-dev] [tempest] Test case for new feature failed in Jenkins check against old release

[Matthew Treinish]

On Thu, Sep 29, 2016 at 03:49:27PM +0800, Bruce Tan wrote:
> Hello everyone,
> 
> I am having a problem writing/updating a test case to verify some new
> feature (in my case, the "description" field for a network).
> 
> Acoording to Tempest Coding Guide[1], I am supposed to check if the
> related feature is there by @test.requires_ext() like this:
> @test.requires_ext(extension="standard-attr-description",
>service="network")
> 
> And according to the same doc,
> > When running a test that requires a certain "feature" in the target
> > cloud, if that feature is missing we should fail, because either the
> > test configuration is invalid, or the cloud is broken and the expected
> > "feature" is not there even if the cloud was configured with it.
> 
> 
> However, my patch[2] got a "-1" from Jenkins because one check
> ("gate-tempest-dsvm-neutron-full-ubuntu-trusty-liberty") failed. The
> reason for failing, I think, is just what I quoted above: the
> tempest.conf[3]  file is configured as
>   [network-feature-enabled]api_extensions = all
> which means any api_extension is supported; but the feature I am
> testing is obviously not there in Liberty, so the API doesn't accept
> "description" field, and the test case failed.
> 
> So my question is, what did I do wrong? Is there some other way
> to skip the case for older releases? Or, maybe we shouldn't use
> "api_extensions=all" (explicitly list all extensions instead, which
> takes some effort obviously) in the configuration file?

You actually did everything correctly for making sure you skip properly
if the new feature isn't present on the tempest side. It's actually a bug in
devstack you're hitting. On stable branches we're supposed to hard code the
extension list of what api extensions are available when we branch the project.
But in the case of liberty, we neglected to do this:

http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest?h=stable%2Fliberty#n430

Instead this should look something like what we did on the kilo branch:

http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest?h=stable/kilo#n413

You just need to push up a patch to devstack's stable/liberty branch that hard
codes the extensions available like we did in previous branches.  Once that's up
add a Depends-On to your commit and it should work fine.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Test case for new feature failed in Jenkins check against old release

[Bruce Tan]

Hello everyone,

I am having a problem writing/updating a test case to verify some new
feature (in my case, the "description" field for a network).

Acoording to Tempest Coding Guide[1], I am supposed to check if the
related feature is there by @test.requires_ext() like this:
@test.requires_ext(extension="standard-attr-description",
   service="network")

And according to the same doc,
> When running a test that requires a certain "feature" in the target
> cloud, if that feature is missing we should fail, because either the
> test configuration is invalid, or the cloud is broken and the expected
> "feature" is not there even if the cloud was configured with it.


However, my patch[2] got a "-1" from Jenkins because one check
("gate-tempest-dsvm-neutron-full-ubuntu-trusty-liberty") failed. The
reason for failing, I think, is just what I quoted above: the
tempest.conf[3]  file is configured as
  [network-feature-enabled]api_extensions = all
which means any api_extension is supported; but the feature I am
testing is obviously not there in Liberty, so the API doesn't accept
"description" field, and the test case failed.

So my question is, what did I do wrong? Is there some other way
to skip the case for older releases? Or, maybe we shouldn't use
"api_extensions=all" (explicitly list all extensions instead, which
takes some effort obviously) in the configuration file?

Thank you in advance.

[1] 
http://docs.openstack.org/developer/tempest/HACKING.html#new-tests-for-new-features

[2] https://review.openstack.org/#/c/374645/
[3]
http://logs.openstack.org/45/374645/2/check/gate-tempest-dsvm-neutron-full-ubuntu-trusty-liberty/31bb416/logs/tempest_conf.txt.gz


--
Bruce Tan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] tempest tests in Horizon GUI

[Masayuki Igawa]

Hi Ofer,

As Andrea said, Tempest shouldn't leave any test resource after a
Tempest run. And Horizon is not mandatory in a Tempest run. But if you
want to verify it, you can do that through your openstack-client or
Horizon dashboard just in case. And you can find your credentials in
your tempest.conf and accounts.yaml file.

If you want to know more details, some documents[1] might be helpful.

[1] 
http://docs.openstack.org/developer/tempest/configuration.html#tempest-configuration
.

Best Regards,
-- Masayuki Igawa


On Thu, Sep 22, 2016 at 7:27 AM, Andrea Frittoli
 wrote:
> Hi Ofer,
>
> Tempest tests try to cleanup after themselves, so in theory after a Tempest
> run you shouldn't see any test resource left around.
>
> If you run your tests using dynamic credentials, test accounts are created
> on the fly, and deleted after they're used. If you use preprovisioned
> credentials, you need to setup a network for them before the test run
> starts.
>
> Andrea
>
>
> On Thu, 22 Sep 2016, 7:02 a.m. Ken'ichi Ohmichi, 
> wrote:
>>
>> Hi Ofer,
>>
>> The scenario test of Horizon has been removed from Tempest since
>> https://review.openstack.org/#/c/313713/
>> And now the test[1] exists in openstack/tempest-horizon.
>> The test is very simple like
>>  * checks that the login page is available
>>  * logs in as a regular user
>>  * checks that the user home page loads without error
>>
>> > When I run a tempest test/scenario-test, should I see the components
>> > (network, subnet, router etc.) in the horizon GUI ?
>>
>> The test doesn't create any resources(network, subnet, etc), so any
>> changes would not happen during the test.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>> [1]:
>> https://github.com/openstack/tempest-horizon/blob/master/tempest_horizon/tests/scenario/test_dashboard_basic_ops.py
>>
>> 2016-09-21 15:01 GMT+02:00 Barber, Ofer :
>> > I have a basic question about tempest.
>> >
>> >
>> >
>> > When I run a tempest test/scenario-test, should I see the components
>> > (network, subnet, router etc.) in the horizon GUI ?
>> >
>> >
>> >
>> > If yes, for what username or what project those are created ?
>> >
>> >
>> >
>> > Thank you,
>> >
>> > Ofer
>> >
>> >
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] tempest tests in Horizon GUI

[Andrea Frittoli]

Hi Ofer,

Tempest tests try to cleanup after themselves, so in theory after a Tempest
run you shouldn't see any test resource left around.

If you run your tests using dynamic credentials, test accounts are created
on the fly, and deleted after they're used. If you use preprovisioned
credentials, you need to setup a network for them before the test run
starts.

Andrea

On Thu, 22 Sep 2016, 7:02 a.m. Ken'ichi Ohmichi, 
wrote:

> Hi Ofer,
>
> The scenario test of Horizon has been removed from Tempest since
> https://review.openstack.org/#/c/313713/
> And now the test[1] exists in openstack/tempest-horizon.
> The test is very simple like
>  * checks that the login page is available
>  * logs in as a regular user
>  * checks that the user home page loads without error
>
> > When I run a tempest test/scenario-test, should I see the components
> (network, subnet, router etc.) in the horizon GUI ?
>
> The test doesn't create any resources(network, subnet, etc), so any
> changes would not happen during the test.
>
> Thanks
> Ken Ohmichi
>
> ---
> [1]:
> https://github.com/openstack/tempest-horizon/blob/master/tempest_horizon/tests/scenario/test_dashboard_basic_ops.py
>
> 2016-09-21 15:01 GMT+02:00 Barber, Ofer :
> > I have a basic question about tempest.
> >
> >
> >
> > When I run a tempest test/scenario-test, should I see the components
> > (network, subnet, router etc.) in the horizon GUI ?
> >
> >
> >
> > If yes, for what username or what project those are created ?
> >
> >
> >
> > Thank you,
> >
> > Ofer
> >
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] tempest tests in Horizon GUI

[Ken'ichi Ohmichi]

Hi Ofer,

The scenario test of Horizon has been removed from Tempest since
https://review.openstack.org/#/c/313713/
And now the test[1] exists in openstack/tempest-horizon.
The test is very simple like
 * checks that the login page is available
 * logs in as a regular user
 * checks that the user home page loads without error

> When I run a tempest test/scenario-test, should I see the components 
> (network, subnet, router etc.) in the horizon GUI ?

The test doesn't create any resources(network, subnet, etc), so any
changes would not happen during the test.

Thanks
Ken Ohmichi

---
[1]: 
https://github.com/openstack/tempest-horizon/blob/master/tempest_horizon/tests/scenario/test_dashboard_basic_ops.py

2016-09-21 15:01 GMT+02:00 Barber, Ofer :
> I have a basic question about tempest.
>
>
>
> When I run a tempest test/scenario-test, should I see the components
> (network, subnet, router etc.) in the horizon GUI ?
>
>
>
> If yes, for what username or what project those are created ?
>
>
>
> Thank you,
>
> Ofer
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest]Tempest test concurrency

[Bob Hansen]

Matthew, this helps tremendously. As you can tell the conclusion I was
heading towards was not accurate.

Now to look a bit deeper.

Thanks,

Bob Hansen
z/VM OpenStack Enablement

Matthew Treinish  wrote on 09/21/2016 11:07:04 AM:

> From: Matthew Treinish 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Date: 09/21/2016 11:09 AM
> Subject: Re: [openstack-dev] [tempest]Tempest test concurrency
>
> On Wed, Sep 21, 2016 at 10:44:51AM -0400, Bob Hansen wrote:
> >
> >
> > I have been looking at some of the stackviz output as I'm trying to
improve
> > the run time of my thrid-party CI. As an example:
> >
> > http://logs.openstack.org/36/371836/1/check/gate-tempest-dsvm-
> full-ubuntu-xenial/087db0f/logs/stackviz/#/stdin/timeline
> >
> > What jumps out is the amount of time that each worker is not running
any
> > tests. I would have expected quite a bit more concurrecy between the
two
> > workers in the chart, e.g. more overlap. I've noticed a simliar thing
with
> > my test runs using 4 workers.
>
> So the gaps between tests aren't actually wait time, the workers
aresaturated
> doing stuff during a run. Those gaps are missing data in the subunit
streams
> that are used as the soure of the data for rendering those timelines. The
gaps
> are where things like setUp, setUpClass, tearDown, tearDownClass, and
> addCleanups which are not added to the subunit stream. It's just an
> artifact of
> the incomplete data, not bad scheduling. This also means that testr does
not
> take into account any of the missing timing when it makes decisions based
on
> previous runs.
>
> >
> > Can anyone explain why this is and where can I find out more
information
> > about the scheduler and what information it is using to decide when to
> > dispatch tests? I'm already feeding my system a prior subunit stream to
> > help influence the scheduler as my test run times are different due to
the
> > way our openstack implementation is architected. A simple round-robin
> > approach is not the most efficeint in my case.
>
> If you're curious about how testr does scheduling most of that happens
here:
>
> https://github.com/testing-cabal/testrepository/blob/master/
> testrepository/testcommand.py
>
> One thing to remember is that testr isn't actually a test runner, it's a
test
> runner runner. It partitions the tests based on time information and
passes
> those to (multiple) test runner workers. The actual order of execution
inside
> those partitions is handled by the test runner itself. (in our case
> subunit.run)
>
> -Matt Treinish
> [attachment "signature.asc" deleted by Bob Hansen/Endicott/IBM]
>
__
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest]Tempest test concurrency

[Matthew Treinish]

On Wed, Sep 21, 2016 at 10:44:51AM -0400, Bob Hansen wrote:
> 
> 
> I have been looking at some of the stackviz output as I'm trying to improve
> the run time of my thrid-party CI. As an example:
> 
> http://logs.openstack.org/36/371836/1/check/gate-tempest-dsvm-full-ubuntu-xenial/087db0f/logs/stackviz/#/stdin/timeline
> 
> What jumps out is the amount of time that each worker is not running any
> tests. I would have expected quite a bit more concurrecy between the two
> workers in the chart, e.g. more overlap. I've noticed a simliar thing with
> my test runs using 4 workers.

So the gaps between tests aren't actually wait time, the workers are saturated
doing stuff during a run. Those gaps are missing data in the subunit streams
that are used as the soure of the data for rendering those timelines. The gaps
are where things like setUp, setUpClass, tearDown, tearDownClass, and
addCleanups which are not added to the subunit stream. It's just an artifact of
the incomplete data, not bad scheduling. This also means that testr does not
take into account any of the missing timing when it makes decisions based on
previous runs.

> 
> Can anyone explain why this is and where can I find out more information
> about the scheduler and what information it is using to decide when to
> dispatch tests? I'm already feeding my system a prior subunit stream to
> help influence the scheduler as my test run times are different due to the
> way our openstack implementation is architected. A simple round-robin
> approach is not the most efficeint in my case.

If you're curious about how testr does scheduling most of that happens here:

https://github.com/testing-cabal/testrepository/blob/master/testrepository/testcommand.py

One thing to remember is that testr isn't actually a test runner, it's a test
runner runner. It partitions the tests based on time information and passes
those to (multiple) test runner workers. The actual order of execution inside
those partitions is handled by the test runner itself. (in our case subunit.run)

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest]Tempest test concurrency

[Bob Hansen]

I have been looking at some of the stackviz output as I'm trying to improve
the run time of my thrid-party CI. As an example:

http://logs.openstack.org/36/371836/1/check/gate-tempest-dsvm-full-ubuntu-xenial/087db0f/logs/stackviz/#/stdin/timeline

What jumps out is the amount of time that each worker is not running any
tests. I would have expected quite a bit more concurrecy between the two
workers in the chart, e.g. more overlap. I've noticed a simliar thing with
my test runs using 4 workers.

Can anyone explain why this is and where can I find out more information
about the scheduler and what information it is using to decide when to
dispatch tests? I'm already feeding my system a prior subunit stream to
help influence the scheduler as my test run times are different due to the
way our openstack implementation is architected. A simple round-robin
approach is not the most efficeint in my case.

(maybe openstack-infra is a better place to ask?)

Thanks!

Bob Hansen
z/VM OpenStack Enablement
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] tempest tests in Horizon GUI

[Barber, Ofer]

I have a basic question about tempest.

When I run a tempest test/scenario-test, should I see the components (network, 
subnet, router etc.) in the horizon GUI ?

If yes, for what username or what project those are created ?

Thank you,
Ofer

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Jay S. Bryant]

Ben,

Thank you very much for the history on the driver here.  This was all 
news to me.  It helps me to understand why cloning was listed as 
supported but was no longer implemented.  Also explains why the simpler 
'cp' based solution hasn't been implemented.  I was wondering why that 
hadn't just been done.


IBM is planning to take over the process of developing/maintaining the 
NFS driver, at least in the near term. Erlon has also been helping with 
this.


We would really like to avoid the driver being removed.  Perhaps I will 
send a separate 'State of the NFS driver' update e-mail to the mailing 
list to explain what I think the state of the driver is and what I see 
to be the plan going forward.


Thanks,

Jay


On 08/25/2016 12:06 PM, Ben Swartzlander wrote:


Originally the NFS driver did support snapshots, but it was 
implemented by just 'cp'ing the file containing the raw bits. This 
works fine (if inefficiently) for unattached volumes, but if you do 
this on an attached volume the snapshot won't be crash consistent at all.


It was decided that we could do better for attached volumes by 
switching to qcow2 and relying on nova to perform the snapshots. Based 
on this, the bad snapshot implementation was removed.


However, for a variety of reasons the nova-assisted snapshot 
implementation has remained unmerged for 2+ years and the NFS driver 
has been an exception to the rules for that whole time.


I would like to see that exception end in the near future with either 
the removal of the driver or the completion of the Nova-assisted 
snapshot implementation, and it doesn't really matter to me which.


There is a 3rd alternative which would be to modify the NFS driver to 
require a specific filesystem that supports snapshots (there are a few 
choices here, but definitely NOT ext4). Unfortunately those of us who 
work for storage vendors aren't motivated to make such a modification 
because it would be effectively creating more competition for 
ourselves. The only way this could happen is if someone not working 
for a storage vendor takes this on.


-Ben

On August 25, 2016 10:39:35 AM Erlon Cruz  wrote:


Hi Jordan, Slade,

Currently NFS driver does not support cloning neither snapshots 
(which are the base for implementing cloning). AFAIC, the NFS driver 
was in Cinder before the minimum requirements being discussed and 
set, so, it just stood there with the features it already supported.


There is currently this job 
'gate-tempest-dsvm-full-devstack-plugin-nfs-nv'[1] that by the way 
are failing in the same test you mentioned tough passing the snapshot 
tests (not shure how the configuration is doing that) and a work[2] 
in progress to support the snapshot feature.


So, Jordan, I think its OK to allow tempest to skip this tests, 
provided that at least in the NFS driver, tempest isn't being an 
enforcement to Cinder minimum features requirements.


Erlon


[1] 
http://logs.openstack.org/86/147186/25/experimental/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/b149960/

[2] https://review.openstack.org/#/c/147186/

On Wed, Aug 24, 2016 at 6:34 PM, Jordan Pittier 
mailto:jordan.pitt...@scality.com>> wrote:



On Wed, Aug 24, 2016 at 6:06 PM, Slade Baumann
mailto:baum...@us.ibm.com>> wrote:

I am attempting to disable clone tests in tempest as they aren't
functioning in NFS. But the tests test_volumes_clone.py and
test_volumes_clone_negative.py don't have the "clone" feature
toggle in them. I thought it obvious that if clone is disabled
in tempest, the tests that simply clone should be disabled.

So I put up a bug and fix for it, but have been talking with
Jordan Pittier and he suggested I come to the mailing list to
get this figured out.

I'm not asking for reviews, unless you want to give them.
I'm simply asking if this is the right way to go about this
or if there is something else I need to do to get this into
Tempest.

Here are the bug and fix:
https://bugs.launchpad.net/tempest/+bug/1615770

https://review.openstack.org/#/c/358813/


I would appreciate any suggestion or direction in this problem.

For extra reference, the clone toggle flag was added here:
https://bugs.launchpad.net/tempest/+bug/1488274


Hi,
Thanks for starting this thread. My point about this patch is, as
"volume clone" is part of the core requirements [1] every Cinder
drive must support, I don't see a need for a feature flag. The
feature flag already exists, but that doesn't mean we should
encourage its usage.

Now, if this really helps the NFS driver (although I don"t know
why we couldn't support clone with NFS)... I don't have a strong
opinion on this patch.

I -1ed th

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Erlon Cruz]

He is mentioning about the Cinder side: https://review.openstack.org/#
/c/147186/

On Fri, Aug 26, 2016 at 7:01 AM, Jordan Pittier 
wrote:

>
>
> On Thu, Aug 25, 2016 at 7:06 PM, Ben Swartzlander 
> wrote:
>
>> Originally the NFS driver did support snapshots, but it was implemented
>> by just 'cp'ing the file containing the raw bits. This works fine (if
>> inefficiently) for unattached volumes, but if you do this on an attached
>> volume the snapshot won't be crash consistent at all.
>>
>> It was decided that we could do better for attached volumes by switching
>> to qcow2 and relying on nova to perform the snapshots. Based on this, the
>> bad snapshot implementation was removed.
>>
>> However, for a variety of reasons the nova-assisted snapshot
>> implementation has remained unmerged for 2+ years and the NFS driver has
>> been an exception to the rules for that whole time.
>>
> I am not sure to understand what you mean by "the nova-assisted snapshot
> implementation has remained unmerged for 2+ years". It looks merged to me
> [1] and several Cinder drivers dependent on it as far as I know.
>
> [1]: http://developer.openstack.org/api-ref-compute-
> v2.1.html#os-assisted-volume-snapshots-v2.1
>
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Jordan Pittier]

On Thu, Aug 25, 2016 at 7:06 PM, Ben Swartzlander 
wrote:

> Originally the NFS driver did support snapshots, but it was implemented by
> just 'cp'ing the file containing the raw bits. This works fine (if
> inefficiently) for unattached volumes, but if you do this on an attached
> volume the snapshot won't be crash consistent at all.
>
> It was decided that we could do better for attached volumes by switching
> to qcow2 and relying on nova to perform the snapshots. Based on this, the
> bad snapshot implementation was removed.
>
> However, for a variety of reasons the nova-assisted snapshot
> implementation has remained unmerged for 2+ years and the NFS driver has
> been an exception to the rules for that whole time.
>
I am not sure to understand what you mean by "the nova-assisted snapshot
implementation has remained unmerged for 2+ years". It looks merged to me
[1] and several Cinder drivers dependent on it as far as I know.

[1]:
http://developer.openstack.org/api-ref-compute-v2.1.html#os-assisted-volume-snapshots-v2.1

-- 
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Slade Baumann]

Ben,In response to your comments, we are working on getting NFS up to date at IBM as we are using it as our storage backend (long story, don't ask lol). A few of use used to focus on Cinder development, so we have a good chance of getting things working. So I agree wholeheartedly that this needs to be resolved and we are working towards that goal (as well as fixing clone, etc).P.S. Thanks for you history of the NFS driver. I didn't know a lot of that.Slade-Ben Swartzlander <b...@swartzlander.org> wrote: -To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org>, Erlon Cruz <sombra...@gmail.com>From: Ben Swartzlander <b...@swartzlander.org>Date: 08/25/2016 12:10PMSubject: Re: [openstack-dev] [tempest][cinder] Clone feature toggle not inclone tests


Originally the NFS driver did
support snapshots, but it was implemented by just 'cp'ing the file
containing the raw bits. This works fine (if inefficiently) for unattached
volumes, but if you do this on an attached volume the snapshot won't be
crash consistent at all.It was decided that we could do
better for attached volumes by switching to qcow2 and relying on nova to
perform the snapshots. Based on this, the bad snapshot implementation was
removed.However, for a variety of
reasons the nova-assisted snapshot implementation has remained unmerged for
2+ years and the NFS driver has been an exception to the rules for that
whole time.I would like to see that
exception end in the near future with either the removal of the driver or
the completion of the Nova-assisted snapshot implementation, and it doesn't
really matter to me which.There is a 3rd alternative
which would be to modify the NFS driver to require a specific filesystem
that supports snapshots (there are a few choices here, but definitely NOT
ext4). Unfortunately those of us who work for storage vendors aren't
motivated to make such a modification because it would be effectively
creating more competition for ourselves. The only way this could happen is
if someone not working for a storage vendor takes this on.-BenOn
August 25, 2016 10:39:35 AM Erlon Cruz <sombra...@gmail.com> wrote:Hi Jordan, Slade,Currently NFS driver
does not support cloning neither snapshots (which are the base for
implementing cloning). AFAIC, the NFS driver was in Cinder before the
minimum requirements being discussed and set, so, it just stood there with
the features it already supported.There is
currently this job
'gate-tempest-dsvm-full-devstack-plugin-nfs-nv'[1] that by the way
are failing in the same test you mentioned tough passing the snapshot tests
(not shure how the configuration is doing that) and a work[2] in progress
to support the snapshot feature.So, Jordan, I
think its OK to allow tempest to skip this tests, provided that at least in
the NFS driver, tempest isn't being an enforcement to Cinder minimum
features
requirements.Erlon[1]
http://logs.openstack.org/86/147186/25/experimental/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/b149960/[2]
https://review.openstack.org/#/c/147186/ On Wed, Aug 24, 2016 at
6:34 PM, Jordan Pittier <jordan.pitt...@scality.com>wrote:On Wed, Aug 24, 2016 at 6:06 PM, Slade Baumann <baum...@us.ibm.com> wrote:I am attempting to
disable clone tests in tempest as they aren'tfunctioning in NFS.
But the tests test_volumes_clone.py andtest_volumes_clone_negative.py
don't have the "clone" featuretoggle in them. I thought
it obvious that if clone is disabledin tempest, the tests that simply
clone should be disabled.So I put up a bug and fix for it, but have
been talking withJordan Pittier and he suggested I come to the mailing
list toget this figured out. I'm not asking for reviews,
unless you want to give them.I'm simply asking if this is the right
way to go about thisor if there is something else I need to do to get
this intoTempest.Here are the bug and fix:https://bugs.launchpad.net/tempest/+bug/1615770https://review.openstack.org/#/c/358813/I
would appreciate any suggestion or direction in this problem.For
extra reference, the clone toggle flag was added here:https://bugs.launchpad.net/tempest/+bug/1488274

Hi, Thanks for starting this
thread. My point about this patch is, as "volume clone" is part
of the core requirements [1] every Cinder drive must support, I don't
see a need for a feature flag. The feature flag already exists, but that
doesn't mean we should encourage its
usage.Now, if this really helps the NFS driver
(although I don"t know why we couldn't support clone with NFS)...
I don't have a strong opinion on this patch.I
-1ed the patch for consistency: I agree that there should be a minimum set
of features expected from a Cinder driver.[1] http://docs.openstack.org/developer/cinder/devref/drivers.html#core-functionalityCheers,Jordan 
_

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Ben Swartzlander]

Originally the NFS driver did support snapshots, but it was implemented by 
just 'cp'ing the file containing the raw bits. This works fine (if 
inefficiently) for unattached volumes, but if you do this on an attached 
volume the snapshot won't be crash consistent at all.


It was decided that we could do better for attached volumes by switching to 
qcow2 and relying on nova to perform the snapshots. Based on this, the bad 
snapshot implementation was removed.


However, for a variety of reasons the nova-assisted snapshot implementation 
has remained unmerged for 2+ years and the NFS driver has been an exception 
to the rules for that whole time.


I would like to see that exception end in the near future with either the 
removal of the driver or the completion of the Nova-assisted snapshot 
implementation, and it doesn't really matter to me which.


There is a 3rd alternative which would be to modify the NFS driver to 
require a specific filesystem that supports snapshots (there are a few 
choices here, but definitely NOT ext4). Unfortunately those of us who work 
for storage vendors aren't motivated to make such a modification because it 
would be effectively creating more competition for ourselves. The only way 
this could happen is if someone not working for a storage vendor takes this on.


-Ben


On August 25, 2016 10:39:35 AM Erlon Cruz  wrote:


Hi Jordan, Slade,

Currently NFS driver does not support cloning neither snapshots (which are
the base for implementing cloning). AFAIC, the NFS driver was in Cinder
before the minimum requirements being discussed and set, so, it just stood
there with the features it already supported.

There is currently this job
'gate-tempest-dsvm-full-devstack-plugin-nfs-nv'[1] that by the way are
failing in the same test you mentioned tough passing the snapshot tests
(not shure how the configuration is doing that) and a work[2] in progress
to support the snapshot feature.

So, Jordan, I think its OK to allow tempest to skip this tests, provided
that at least in the NFS driver, tempest isn't being an enforcement to
Cinder minimum features requirements.

Erlon


[1]
http://logs.openstack.org/86/147186/25/experimental/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/b149960/
[2] https://review.openstack.org/#/c/147186/

On Wed, Aug 24, 2016 at 6:34 PM, Jordan Pittier 
wrote:



On Wed, Aug 24, 2016 at 6:06 PM, Slade Baumann  wrote:


I am attempting to disable clone tests in tempest as they aren't
functioning in NFS. But the tests test_volumes_clone.py and
test_volumes_clone_negative.py don't have the "clone" feature
toggle in them. I thought it obvious that if clone is disabled
in tempest, the tests that simply clone should be disabled.

So I put up a bug and fix for it, but have been talking with
Jordan Pittier and he suggested I come to the mailing list to
get this figured out.

I'm not asking for reviews, unless you want to give them.
I'm simply asking if this is the right way to go about this
or if there is something else I need to do to get this into
Tempest.

Here are the bug and fix:
https://bugs.launchpad.net/tempest/+bug/1615770
https://review.openstack.org/#/c/358813/

I would appreciate any suggestion or direction in this problem.

For extra reference, the clone toggle flag was added here:
https://bugs.launchpad.net/tempest/+bug/1488274

Hi,

Thanks for starting this thread. My point about this patch is, as "volume
clone" is part of the core requirements [1] every Cinder drive must
support, I don't see a need for a feature flag. The feature flag already
exists, but that doesn't mean we should encourage its usage.

Now, if this really helps the NFS driver (although I don"t know why we
couldn't support clone with NFS)... I don't have a strong opinion on this
patch.

I -1ed the patch for consistency: I agree that there should be a minimum
set of features expected from a Cinder driver.

[1] http://docs.openstack.org/developer/cinder/devref/drivers.html#core-
functionality

Cheers,
Jordan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Erlon Cruz]

Hi Jordan, Slade,

Currently NFS driver does not support cloning neither snapshots (which are
the base for implementing cloning). AFAIC, the NFS driver was in Cinder
before the minimum requirements being discussed and set, so, it just stood
there with the features it already supported.

There is currently this job
'gate-tempest-dsvm-full-devstack-plugin-nfs-nv'[1] that by the way are
failing in the same test you mentioned tough passing the snapshot tests
(not shure how the configuration is doing that) and a work[2] in progress
to support the snapshot feature.

So, Jordan, I think its OK to allow tempest to skip this tests, provided
that at least in the NFS driver, tempest isn't being an enforcement to
Cinder minimum features requirements.

Erlon


[1]
http://logs.openstack.org/86/147186/25/experimental/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/b149960/
[2] https://review.openstack.org/#/c/147186/

On Wed, Aug 24, 2016 at 6:34 PM, Jordan Pittier 
wrote:

>
> On Wed, Aug 24, 2016 at 6:06 PM, Slade Baumann  wrote:
>
>> I am attempting to disable clone tests in tempest as they aren't
>> functioning in NFS. But the tests test_volumes_clone.py and
>> test_volumes_clone_negative.py don't have the "clone" feature
>> toggle in them. I thought it obvious that if clone is disabled
>> in tempest, the tests that simply clone should be disabled.
>>
>> So I put up a bug and fix for it, but have been talking with
>> Jordan Pittier and he suggested I come to the mailing list to
>> get this figured out.
>>
>> I'm not asking for reviews, unless you want to give them.
>> I'm simply asking if this is the right way to go about this
>> or if there is something else I need to do to get this into
>> Tempest.
>>
>> Here are the bug and fix:
>> https://bugs.launchpad.net/tempest/+bug/1615770
>> https://review.openstack.org/#/c/358813/
>>
>> I would appreciate any suggestion or direction in this problem.
>>
>> For extra reference, the clone toggle flag was added here:
>> https://bugs.launchpad.net/tempest/+bug/1488274
>>
>> Hi,
> Thanks for starting this thread. My point about this patch is, as "volume
> clone" is part of the core requirements [1] every Cinder drive must
> support, I don't see a need for a feature flag. The feature flag already
> exists, but that doesn't mean we should encourage its usage.
>
> Now, if this really helps the NFS driver (although I don"t know why we
> couldn't support clone with NFS)... I don't have a strong opinion on this
> patch.
>
> I -1ed the patch for consistency: I agree that there should be a minimum
> set of features expected from a Cinder driver.
>
> [1] http://docs.openstack.org/developer/cinder/devref/drivers.html#core-
> functionality
>
> Cheers,
> Jordan
>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Jordan Pittier]

On Wed, Aug 24, 2016 at 6:06 PM, Slade Baumann  wrote:

> I am attempting to disable clone tests in tempest as they aren't
> functioning in NFS. But the tests test_volumes_clone.py and
> test_volumes_clone_negative.py don't have the "clone" feature
> toggle in them. I thought it obvious that if clone is disabled
> in tempest, the tests that simply clone should be disabled.
>
> So I put up a bug and fix for it, but have been talking with
> Jordan Pittier and he suggested I come to the mailing list to
> get this figured out.
>
> I'm not asking for reviews, unless you want to give them.
> I'm simply asking if this is the right way to go about this
> or if there is something else I need to do to get this into
> Tempest.
>
> Here are the bug and fix:
> https://bugs.launchpad.net/tempest/+bug/1615770
> https://review.openstack.org/#/c/358813/
>
> I would appreciate any suggestion or direction in this problem.
>
> For extra reference, the clone toggle flag was added here:
> https://bugs.launchpad.net/tempest/+bug/1488274
>
> Hi,
Thanks for starting this thread. My point about this patch is, as "volume
clone" is part of the core requirements [1] every Cinder drive must
support, I don't see a need for a feature flag. The feature flag already
exists, but that doesn't mean we should encourage its usage.

Now, if this really helps the NFS driver (although I don"t know why we
couldn't support clone with NFS)... I don't have a strong opinion on this
patch.

I -1ed the patch for consistency: I agree that there should be a minimum
set of features expected from a Cinder driver.

[1]
http://docs.openstack.org/developer/cinder/devref/drivers.html#core-functionality

Cheers,
Jordan

-- 
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][cinder] Clone feature toggle not in clone tests

[Slade Baumann]

I am attempting to disable clone tests in tempest as they aren'tfunctioning in NFS. But the tests test_volumes_clone.py andtest_volumes_clone_negative.py don't have the "clone" featuretoggle in them. I thought it obvious that if clone is disabledin tempest, the tests that simply clone should be disabled.So I put up a bug and fix for it, but have been talking withJordan Pittier and he suggested I come to the mailing list toget this figured out. I'm not asking for reviews, unless you want to give them.I'm simply asking if this is the right way to go about thisor if there is something else I need to do to get this intoTempest.Here are the bug and fix:https://bugs.launchpad.net/tempest/+bug/1615770https://review.openstack.org/#/c/358813/I would appreciate any suggestion or direction in this problem.For extra reference, the clone toggle flag was added here:https://bugs.launchpad.net/tempest/+bug/1488274


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][plugins] Service clients stable interface in tempest.lib

[Andrea Frittoli]

Hi Tempest plugin developers / maintainers,

a new stable interface "ServiceClients" [0][1] is available in Tempest that
provides tests with a convenient way to access all available service
clients.
At the same time the plugin interface is extended with an optional method
"get_service_clients" [2][3], that allows plugins to declare any service
client they implement.

When a plugin uses "ServiceClients", all Tempest stable clients and all
clients exposed by installed plugins via the new interface will be
automatically available and pre-configured.
This makes it easy for plugins to access service clients and to write
integration tests which access APIs via service clients implemented in
different plugins.

One caveat is that three of the Tempest service clients are stable at the
moment: compute, network and image.
Work is in progress to make the other three (identity, volume and
object-storage) stable as well.

ServiceClients replaces the legacy "Manager" classes [2] and [3], which are
not stable interface and may change or disappear without prior notice.
We'll keep them both around unchanged for a reasonable amount of time to
allow for plugins to switch to the new interface - and at least as long as
the remaining Tempest service clients are available in tempest.lib.

I tried out the new interface with a couple of plugins already [6][7]. I
would heartily recommend migrating your Tempest plugin to the new
interface, especially if your currently using one of the unstable "Manager"
interfaces today. Please reach out for help in the #openstack-qa room if
you have questions on the new interfaces, trouble using them in your
plugin, requests for features or feedback.

Thanks for reading through.

Andrea Frittoli (andreaf)

[0]
http://docs.openstack.org/developer/tempest/library/clients.html#tempest.lib.services.clients.ServiceClients

[1]
http://docs.openstack.org/releasenotes/tempest/unreleased.html#new-features
[2] http://docs.openstack.org/developer/tempest/plugin.html#service-clients
[3]
http://docs.openstack.org/releasenotes/tempest/unreleased.html#new-features
[4]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/manager.py#n26
[5]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/clients.py#n35
[6] https://review.openstack.org/#/c/334596/
[7] https://review.openstack.org/#/c/338486/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][swift][radosgw] Can we please merge the fix for the RFC 7230 violation issue?

[John Dickinson]

On 8 Aug 2016, at 9:50, Jay Pipes wrote:

> Tempest devs,
>
> Let me please draw your attention to a LP bug that may not seem particularly 
> high priority, but I believe could be resolved easily with a patch already 
> proposed.
>
> LP bug 1536251 [1] accurately states that Tempest is actively verifying that 
> an OpenStack API call violates RFC 7230.
>
> When a 204 No Content is received, the Content-Length header MUST NOT be 
> present.
>
> However, Swift returns a Content-Length header and also an HTTP response code 
> of 204 for a request to list containers of a new user (that has no 
> containers).
>
> Tempest has been validating this behaviour even though it is a violation of 
> RFC 7230:
>
> https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_account_services.py#L81
>
> RadosGW provides a proxy API that attempts to match the OpenStack Object 
> Storage API, backed by Ceph object storage. In order for RadosGW to pass 
> RefStack's burden of compatibility, it must pass the Tempest OpenStack Object 
> Storage API tests. It currently cannot do so because RadosGW does not violate 
> RFC 7230.
>
> The RadosGW developer community does not wish to argue about whether or not 
> to make Swift's API comply with RFC 7230. At the same time, they do not want 
> to add a configuration option to RadosGW to force the proxy service to 
> violate RFC 7230 just to satisfy the RefStack/Tempest API tests.

I think tempest should merge the proposed patch (or one like it) so that 
content-length isn't checked on a 204 response. That solves the discrepancy 
when deployers are running Swift behind some other web server or caching system 
that is stripping the header. On the Swift side, we've got to consider the 
question of risk of breaking clients vs violating updated RFCs. My gut says 
we'll drip the header, but that's something that we'll discuss. But either way, 
it seems silly to me to be blocked here by tempest.


>
> Therefore, Radoslaw (cc'd) has proposed a patch to Tempest that would allow 
> RadosGW's proxy API to meet the RefStack compatibility tests while also not 
> violating RFC 7230 and not requiring any change of Swift:
>
> https://review.openstack.org/#/c/272062
>
> I ask Tempest devs to re-review the above patch and consider merging it for 
> the sake of collaboration between the OpenStack and Ceph developer 
> communities.
>
> Thanks very much!
> -jay
>
> [1] https://bugs.launchpad.net/tempest/+bug/1536251
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][swift][radosgw] Can we please merge the fix for the RFC 7230 violation issue?

[Dave Walker]

On 8 August 2016 at 17:50, Jay Pipes  wrote:

> Tempest devs,
>
> Let me please draw your attention to a LP bug that may not seem
> particularly high priority, but I believe could be resolved easily with a
> patch already proposed.
>
> LP bug 1536251 [1] accurately states that Tempest is actively verifying
> that an OpenStack API call violates RFC 7230.
>
> When a 204 No Content is received, the Content-Length header MUST NOT be
> present.
>
> However, Swift returns a Content-Length header and also an HTTP response
> code of 204 for a request to list containers of a new user (that has no
> containers).
>
> Tempest has been validating this behaviour even though it is a violation
> of RFC 7230:
>
> https://github.com/openstack/tempest/blob/master/tempest/api
> /object_storage/test_account_services.py#L81
>
> RadosGW provides a proxy API that attempts to match the OpenStack Object
> Storage API, backed by Ceph object storage. In order for RadosGW to pass
> RefStack's burden of compatibility, it must pass the Tempest OpenStack
> Object Storage API tests. It currently cannot do so because RadosGW does
> not violate RFC 7230.
>
> The RadosGW developer community does not wish to argue about whether or
> not to make Swift's API comply with RFC 7230. At the same time, they do not
> want to add a configuration option to RadosGW to force the proxy service to
> violate RFC 7230 just to satisfy the RefStack/Tempest API tests.
>
> Therefore, Radoslaw (cc'd) has proposed a patch to Tempest that would
> allow RadosGW's proxy API to meet the RefStack compatibility tests while
> also not violating RFC 7230 and not requiring any change of Swift:
>
> https://review.openstack.org/#/c/272062
>
> I ask Tempest devs to re-review the above patch and consider merging it
> for the sake of collaboration between the OpenStack and Ceph developer
> communities.
>
> Thanks very much!
> -jay
>
> [1] https://bugs.launchpad.net/tempest/+bug/1536251



These sorts of issues aren't just theoretical and following policy for the
sake of it..  Glance had 3 x areas where 200 responses that also included a
Location header (against RFC-2616 §14.30) which totally broke glance when
deployed behind apache+fcgid+flup (the presence of Location, that stack
rewrote it to a 302).

Fun bug btw:
https://bugs.launchpad.net/glance/+bug/1299095

--
Kind Regards,
Dave Walker
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][swift][radosgw] Can we please merge the fix for the RFC 7230 violation issue?

[Jay Pipes]

Tempest devs,

Let me please draw your attention to a LP bug that may not seem 
particularly high priority, but I believe could be resolved easily with 
a patch already proposed.


LP bug 1536251 [1] accurately states that Tempest is actively verifying 
that an OpenStack API call violates RFC 7230.


When a 204 No Content is received, the Content-Length header MUST NOT be 
present.


However, Swift returns a Content-Length header and also an HTTP response 
code of 204 for a request to list containers of a new user (that has no 
containers).


Tempest has been validating this behaviour even though it is a violation 
of RFC 7230:


https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_account_services.py#L81

RadosGW provides a proxy API that attempts to match the OpenStack Object 
Storage API, backed by Ceph object storage. In order for RadosGW to pass 
RefStack's burden of compatibility, it must pass the Tempest OpenStack 
Object Storage API tests. It currently cannot do so because RadosGW does 
not violate RFC 7230.


The RadosGW developer community does not wish to argue about whether or 
not to make Swift's API comply with RFC 7230. At the same time, they do 
not want to add a configuration option to RadosGW to force the proxy 
service to violate RFC 7230 just to satisfy the RefStack/Tempest API tests.


Therefore, Radoslaw (cc'd) has proposed a patch to Tempest that would 
allow RadosGW's proxy API to meet the RefStack compatibility tests while 
also not violating RFC 7230 and not requiring any change of Swift:


https://review.openstack.org/#/c/272062

I ask Tempest devs to re-review the above patch and consider merging it 
for the sake of collaboration between the OpenStack and Ceph developer 
communities.


Thanks very much!
-jay

[1] https://bugs.launchpad.net/tempest/+bug/1536251

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova] Microversions support coverage

[Timofei Durakov]

I think we could at least fix-up schema validation.

On Wed, Jul 6, 2016 at 5:51 PM, Matt Riedemann 
wrote:

> On 7/6/2016 9:37 AM, Timofei Durakov wrote:
>
>> Hi,
>>
>> there are several patches in a tempest that improves micro
>> versions coverage for Nova REST API:
>> https://review.openstack.org/#/c/337598
>> https://review.openstack.org/#/c/338248
>> https://review.openstack.org/#/c/287605
>> https://review.openstack.org/#/c/338256
>> https://review.openstack.org/#/c/233176
>> https://review.openstack.org/#/c/327191
>>
>> The purpose of this thread is to align our efforts in merging these and
>> also ask other contributors to join this activity, as it will improve
>> nova tempest coverage.
>>
>>
>> Timofey.
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> Are these all needed in Tempest? We don't need 100% nova microversion REST
> API coverage in Tempest if there are things that we can test inside nova,
> like in the functional tests. The only thing we don't have in nova is the
> response schema validation. For the most part I only ask for a microversion
> test in Tempest when it requires other services like cinder or neutron, or
> it hits new behavior in the virt drivers.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova] Microversions support coverage

[Matt Riedemann]

On 7/6/2016 9:37 AM, Timofei Durakov wrote:

Hi,

there are several patches in a tempest that improves micro
versions coverage for Nova REST API:
https://review.openstack.org/#/c/337598
https://review.openstack.org/#/c/338248
https://review.openstack.org/#/c/287605
https://review.openstack.org/#/c/338256
https://review.openstack.org/#/c/233176
https://review.openstack.org/#/c/327191

The purpose of this thread is to align our efforts in merging these and
also ask other contributors to join this activity, as it will improve
nova tempest coverage.


Timofey.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Are these all needed in Tempest? We don't need 100% nova microversion 
REST API coverage in Tempest if there are things that we can test inside 
nova, like in the functional tests. The only thing we don't have in nova 
is the response schema validation. For the most part I only ask for a 
microversion test in Tempest when it requires other services like cinder 
or neutron, or it hits new behavior in the virt drivers.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][nova] Microversions support coverage

[Timofei Durakov]

Hi,

there are several patches in a tempest that improves micro
versions coverage for Nova REST API:
https://review.openstack.org/#/c/337598
https://review.openstack.org/#/c/338248
https://review.openstack.org/#/c/287605
https://review.openstack.org/#/c/338256
https://review.openstack.org/#/c/233176
https://review.openstack.org/#/c/327191

The purpose of this thread is to align our efforts in merging these and
also ask other contributors to join this activity, as it will improve nova
tempest coverage.


Timofey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Arkady_Kanevsky]

There is a version of Tempest that is released as part of OpenStack release.
Agree with Mark that we should stick to versions parity.

-Original Message-
From: Mark Voelker [mailto:mvoel...@vmware.com]
Sent: Monday, June 20, 2016 8:25 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable 
some strict response checking for interop testing


> On Jun 20, 2016, at 8:46 AM, Doug Hellmann wrote:
>
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
>
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>>
>>
>>> I don't think DefCore actually needs to change old versions of
>>> Tempest, but maybe Chris or Mark can verify that?
>>
>> So if I'm groking this correctly, there's kind of two scenarios being
>> painted here. One is the "LCD" approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline. The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline. The
>> former may result in some fairly long-lived flags, and the latter is
>> actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>>
>> In some cases, tests get flagged in the Guidelines because of bugs in
>> the test or because the test needs refactoring. The underlying
>> Capabilities the those tests are testing actually work fine. Once we
>> identify such an issue, the test can be fixed...in master. Under the
>> first scenario, this potentially creates some very long-lived flags:
>>
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released). It's one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud. They run the Juno EOL version of Tempest (tag=8), they
>> find a test issue, and we flag it. A few weeks later, a fix lands in
>> Tempest. Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest. That doesn't have the
>> fix, so the flag stays. Another six months goes by and we get a
>> Guideline and we're up to the Liberty-EOL version of Tempest. No
>> fix, flag stays. Six more months, and now we're at Mitaka-EOL, and
>> that's the first version that includes the fix.
>>
>> Generally speaking long lived flags aren't so great because it means
>> the tests are not required...which means there's less or no assurance
>> that the capabilities they test for actually work in the clouds that
>> adhere to those Guidelines. So, the worst-case scenario here looks
>> kind of ugly.
>>
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API's that are long-lived
>> across many releases, so we haven't run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date. In fact
>> I'm struggling to think of a time we've flagged something because
>> someone complained the test wasn't runnable against an older release
>> covered by the Guideline in question. I can think of plenty of times
>> where we've flagged something due to a test issue though...keep in mind
>> we're still in pretty formative times with DefCore here where these
>> tests are starting to be used in a new way for the first time.
>> Anyway, as Matt points out we could potentially use a much newer
>> Tempest tag: tag=11 (which is the start of Newton development and is
>> a roughly 2 month old version of Tempest). Next Guideline rolls
>> around, we use the tag for start-of-ocata, and we get the fix and can
>> drop the flag.
>>
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past, and
>> still can). When we see a fix for a flagged test go in, we or the
>> Refstack folks can do a quick test to make sure everything's in order
>> and then update that SHA to match the version with the fix. That way
>> we're relatively sure we have a version that works today, and will
>> work when we drop the flag in the next Guideline too. When we
>> finalize that next Guideline, we also update the test-repositories
>> section of the n

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Arkady_Kanevsky]

Chris,
If we add openstack config to refstack submission will that provide sufficient 
info for "interoperability" LOGO?
That includes version  of APIs for each service.
https://review.openstack.org/#/c/300057/

Thanks,
Arkady

-Original Message-
From: Chris Hoge [mailto:ch...@openstack.org]
Sent: Wednesday, June 22, 2016 1:37 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tempest][nova][defcore] Add option to disable 
some strict response checking for interop testing


> On Jun 22, 2016, at 11:24 AM, Sean Dague wrote:
>
> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>>
>>> On Jun 20, 2016, at 5:10 AM, Sean Dague
>>> > wrote:
>>>
>>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>>>
>>>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe
>>>>> > wrote:
>>>>>
>>>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish
>>>>> > wrote:
>>>>>
>>>>>> But, if we add another possible state on the defcore side like
>>>>>> conditional pass, warning, yellow, etc. (the name doesn't matter)
>>>>>> which is used to indicate that things on product X could only
>>>>>> pass when strict validation was disabled (and be clear about
>>>>>> where and why) then my concerns would be alleviated.
>>>>>> I just do
>>>>>> not want this to end up not being visible to end users trying to
>>>>>> evaluate interoperability of different clouds using the test
>>>>>> results.
>>>>>
>>>>> +1
>>>>>
>>>>> Don't fail them, but don't cover up their incompatibility, either.
>>>>> -- Ed Leafe
>>>>
>>>> That's not my proposal. My requirement is that vendors who want to
>>>> do this state exactly which APIs are sending back additional data,
>>>> and that this information be published.
>>>>
>>>> There are different levels of incompatibility. A response with
>>>> additional data that can be safely ignored is different from a
>>>> changed response that would cause a client to fail.
>>>
>>> It's actually not different. It's really not.
>>>
>>> This idea that it's safe to add response data is based on an
>>> assumption that software versions only move forward. If you have a
>>> single deploy of software, that's fine.
>>>
>>> However as noted, we've got production clouds on Juno <-> Mitaka in
>>> the wild. Which means if we want to support horizontal transfer
>>> between clouds, the user experienced timeline might be start on a
>>> Mitaka cloud, then try to move to Juno. So anything added from Juno
>>> -> Mitaka without signaling has exactly the same client breaking
>>> behavior as removing attributes.
>>>
>>> Which is why microversions are needed for attribute adds.
>>
>> I'd like to note that Nova v2.0 is still a supported API, which as
>> far as I understand allows for additional attributes and extensions.
>> That Tempest doesn't allow for disabling strict checking when using a
>> v2.0 endpoint is a problem.
>>
>> The reporting of v2.0 in the Marketplace (which is what we do right
>> now) is also a signal to a user that there may be vendor additions to
>> the API.
>>
>> DefCore doesn't disallow the use of a 2.0 endpoint as part of the
>> interoperability standard.
>
> This is a point of confusion.
>
> The API definition did not allow that. The implementation of the API
> stack did.

And downstream vendors took advantage of that. We may not like it, but it's a 
reality in the current ecosystem.

> In Liberty the v2.0 API is optionally provided by a different backend
> stack that doesn't support extensions.
> In Mitaka it is default v2.0 API on a non extensions backend In Newton
> the old backend is deleted.
>
> From Newton forward there is still a v2.0 API, but all the code hooks
> that provided facilities for extensions are gone.

It's really important that the current documentation reflect the code and 
intent of the dev team. As of writing this e-mail,

"* v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will be deprecated in the near 
future.)"[1]

Even with this being removed in Newton, DefCore still has to allow for it in 
every supported version.

-Chris

[1] http://docs.openstack.org/developer/nova/

> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 03:13 PM, Jay Faulkner wrote:
> 
> 
> On 6/22/16 12:01 PM, Sean Dague wrote:
>> On 06/22/2016 02:37 PM, Chris Hoge wrote:
 On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:

 On 06/22/2016 01:59 PM, Chris Hoge wrote:
>> On Jun 20, 2016, at 5:10 AM, Sean Dague > > wrote:
>>
>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
 On Jun 14, 2016, at 3:59 PM, Edward Leafe >>> > wrote:

 On Jun 14, 2016, at 5:50 PM, Matthew Treinish >>> > wrote:

> But, if we add another possible state on the defcore side like
> conditional pass,
> warning, yellow, etc. (the name doesn't matter) which is used to
> indicate that
> things on product X could only pass when strict validation was
> disabled (and
> be clear about where and why) then my concerns would be alleviated.
> I just do
> not want this to end up not being visible to end users trying to
> evaluate
> interoperability of different clouds using the test results.
 +1

 Don't fail them, but don't cover up their incompatibility, either.
 -- Ed Leafe
>>> That’s not my proposal. My requirement is that vendors who want to do
>>> this
>>> state exactly which APIs are sending back additional data, and that this
>>> information be published.
>>>
>>> There are different levels of incompatibility. A response with
>>> additional data
>>> that can be safely ignored is different from a changed response that
>>> would
>>> cause a client to fail.
>> It's actually not different. It's really not.
>>
>> This idea that it's safe to add response data is based on an assumption
>> that software versions only move forward. If you have a single deploy of
>> software, that's fine.
>>
>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>> wild. Which means if we want to support horizontal transfer between
>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>> then try to move to Juno. So anything added from Juno -> Mitaka without
>> signaling has exactly the same client breaking behavior as removing
>> attributes.
>>
>> Which is why microversions are needed for attribute adds.
> I’d like to note that Nova v2.0 is still a supported API, which
> as far as I understand allows for additional attributes and
> extensions. That Tempest doesn’t allow for disabling strict
> checking when using a v2.0 endpoint is a problem.
>
> The reporting of v2.0 in the Marketplace (which is what we do
> right now) is also a signal to a user that there may be vendor
> additions to the API.
>
> DefCore doesn’t disallow the use of a 2.0 endpoint as part
> of the interoperability standard.
 This is a point of confusion.

 The API definition did not allow that. The implementation of the API
 stack did.
>>> And downstream vendors took advantage of that. We may
>>> not like it, but it’s a reality in the current ecosystem.
>> And we started saying "stop it" 2 years ago. And we've consistently been
>> saying stop it all along. And now it's gone.
>>
>> And yes, for people that did not get ahead of this issue and engage the
>> community, it now hurts. But this has been a quite long process.
> I don't wanna wade fully into this discussion, but a question about this
> here as there seems to be somewhat of a double standard. I know
> upstream, we generally "pay the price" for bad API decisions almost
> indefinitely, because we don't want to break users. 

Upstream pays the upstream API cost. Because that's what's in upstream
and because that's what the API is defined as.

> Is it reasonable to
> expecting a public/vendor cloud, who will typically has even more
> change-averse users, to change that API out from under users without a
> version bump?

What you are asking is that upstream pays the fork cost of every public
cloud forever because they didn't collaborate upstream for their needs.

That can't be the right answer in Open Source, ever. It immediately
makes upstream actually impossible to fix root issues. Like security
issues, which is a huge reason that we put the strict inbound validation
into the API stack, as it means the data we're acting on has a
sanitization layer.

If your feature is not in tree, and part of the upstream testing, it is
at risk of break at any time. Upstream is not just a salt mine you
extract free stuff from, it's a garden you need to participate in and
sow and harvest together.

Which, again, is why we've been having this conversation for *2 years*
in the open. There was a 2 year coasting window to get there, to propose
the required elements into upstream, and have a completely soft landing.
People didn't do that. Why, I don't know.

 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Jay Faulkner]

On 6/22/16 12:01 PM, Sean Dague wrote:
> On 06/22/2016 02:37 PM, Chris Hoge wrote:
>>> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
>>>
>>> On 06/22/2016 01:59 PM, Chris Hoge wrote:
> On Jun 20, 2016, at 5:10 AM, Sean Dague  > wrote:
>
> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe >> > wrote:
>>>
>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish >> > wrote:
>>>
 But, if we add another possible state on the defcore side like
 conditional pass,
 warning, yellow, etc. (the name doesn't matter) which is used to
 indicate that
 things on product X could only pass when strict validation was
 disabled (and
 be clear about where and why) then my concerns would be alleviated.
 I just do
 not want this to end up not being visible to end users trying to
 evaluate
 interoperability of different clouds using the test results.
>>> +1
>>>
>>> Don't fail them, but don't cover up their incompatibility, either.
>>> -- Ed Leafe
>> That’s not my proposal. My requirement is that vendors who want to do
>> this
>> state exactly which APIs are sending back additional data, and that this
>> information be published.
>>
>> There are different levels of incompatibility. A response with
>> additional data
>> that can be safely ignored is different from a changed response that
>> would
>> cause a client to fail.
> It's actually not different. It's really not.
>
> This idea that it's safe to add response data is based on an assumption
> that software versions only move forward. If you have a single deploy of
> software, that's fine.
>
> However as noted, we've got production clouds on Juno <-> Mitaka in the
> wild. Which means if we want to support horizontal transfer between
> clouds, the user experienced timeline might be start on a Mitaka cloud,
> then try to move to Juno. So anything added from Juno -> Mitaka without
> signaling has exactly the same client breaking behavior as removing
> attributes.
>
> Which is why microversions are needed for attribute adds.
 I’d like to note that Nova v2.0 is still a supported API, which
 as far as I understand allows for additional attributes and
 extensions. That Tempest doesn’t allow for disabling strict
 checking when using a v2.0 endpoint is a problem.

 The reporting of v2.0 in the Marketplace (which is what we do
 right now) is also a signal to a user that there may be vendor
 additions to the API.

 DefCore doesn’t disallow the use of a 2.0 endpoint as part
 of the interoperability standard.
>>> This is a point of confusion.
>>>
>>> The API definition did not allow that. The implementation of the API
>>> stack did.
>> And downstream vendors took advantage of that. We may
>> not like it, but it’s a reality in the current ecosystem.
> And we started saying "stop it" 2 years ago. And we've consistently been
> saying stop it all along. And now it's gone.
>
> And yes, for people that did not get ahead of this issue and engage the
> community, it now hurts. But this has been a quite long process.
I don't wanna wade fully into this discussion, but a question about this
here as there seems to be somewhat of a double standard. I know
upstream, we generally "pay the price" for bad API decisions almost
indefinitely, because we don't want to break users. Is it reasonable to
expecting a public/vendor cloud, who will typically has even more
change-averse users, to change that API out from under users without a
version bump?

-Jay
>>> In Liberty the v2.0 API is optionally provided by a different backend
>>> stack that doesn't support extensions.
>>> In Mitaka it is default v2.0 API on a non extensions backend
>>> In Newton the old backend is deleted.
>>>
>>> From Newton forward there is still a v2.0 API, but all the code hooks
>>> that provided facilities for extensions are gone.
>> It’s really important that the current documentation reflect the
>> code and intent of the dev team. As of writing this e-mail, 
>>
>> "• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
>> be deprecated in the near future.)”[1]
>>
>> Even with this being removed in Newton, DefCore still has
>> to allow for it in every supported version.
> The v2 extensions link there, you will notice, is upstream extensions.
> All of which default on for the new code stack.
>
> Everything documented there still works on the new code stack. The v2 +
> v2 extensions linked there remains supported in Newton.
>
> The wording on this page should be updated, it is in the Nova developer
> docs, intended for people working on Nova upstream. They lag a bit from
> where reality is, as does documentation everywhere.
>
>   -Sean
>




signature

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 02:37 PM, Chris Hoge wrote:
> 
>> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
>>
>> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>>>
 On Jun 20, 2016, at 5:10 AM, Sean Dague >>> > wrote:

 On 06/14/2016 07:19 PM, Chris Hoge wrote:
>
>> On Jun 14, 2016, at 3:59 PM, Edward Leafe > > wrote:
>>
>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish > > wrote:
>>
>>> But, if we add another possible state on the defcore side like
>>> conditional pass,
>>> warning, yellow, etc. (the name doesn't matter) which is used to
>>> indicate that
>>> things on product X could only pass when strict validation was
>>> disabled (and
>>> be clear about where and why) then my concerns would be alleviated.
>>> I just do
>>> not want this to end up not being visible to end users trying to
>>> evaluate
>>> interoperability of different clouds using the test results.
>>
>> +1
>>
>> Don't fail them, but don't cover up their incompatibility, either.
>> -- Ed Leafe
>
> That’s not my proposal. My requirement is that vendors who want to do
> this
> state exactly which APIs are sending back additional data, and that this
> information be published.
>
> There are different levels of incompatibility. A response with
> additional data
> that can be safely ignored is different from a changed response that
> would
> cause a client to fail.

 It's actually not different. It's really not.

 This idea that it's safe to add response data is based on an assumption
 that software versions only move forward. If you have a single deploy of
 software, that's fine.

 However as noted, we've got production clouds on Juno <-> Mitaka in the
 wild. Which means if we want to support horizontal transfer between
 clouds, the user experienced timeline might be start on a Mitaka cloud,
 then try to move to Juno. So anything added from Juno -> Mitaka without
 signaling has exactly the same client breaking behavior as removing
 attributes.

 Which is why microversions are needed for attribute adds.
>>>
>>> I’d like to note that Nova v2.0 is still a supported API, which
>>> as far as I understand allows for additional attributes and
>>> extensions. That Tempest doesn’t allow for disabling strict
>>> checking when using a v2.0 endpoint is a problem.
>>>
>>> The reporting of v2.0 in the Marketplace (which is what we do
>>> right now) is also a signal to a user that there may be vendor
>>> additions to the API.
>>>
>>> DefCore doesn’t disallow the use of a 2.0 endpoint as part
>>> of the interoperability standard.
>>
>> This is a point of confusion.
>>
>> The API definition did not allow that. The implementation of the API
>> stack did.
> 
> And downstream vendors took advantage of that. We may
> not like it, but it’s a reality in the current ecosystem.

And we started saying "stop it" 2 years ago. And we've consistently been
saying stop it all along. And now it's gone.

And yes, for people that did not get ahead of this issue and engage the
community, it now hurts. But this has been a quite long process.

>> In Liberty the v2.0 API is optionally provided by a different backend
>> stack that doesn't support extensions.
>> In Mitaka it is default v2.0 API on a non extensions backend
>> In Newton the old backend is deleted.
>>
>> From Newton forward there is still a v2.0 API, but all the code hooks
>> that provided facilities for extensions are gone.
> 
> It’s really important that the current documentation reflect the
> code and intent of the dev team. As of writing this e-mail, 
> 
> "• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
> be deprecated in the near future.)”[1]
> 
> Even with this being removed in Newton, DefCore still has
> to allow for it in every supported version.

The v2 extensions link there, you will notice, is upstream extensions.
All of which default on for the new code stack.

Everything documented there still works on the new code stack. The v2 +
v2 extensions linked there remains supported in Newton.

The wording on this page should be updated, it is in the Nova developer
docs, intended for people working on Nova upstream. They lag a bit from
where reality is, as does documentation everywhere.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 22, 2016, at 11:24 AM, Sean Dague  wrote:
> 
> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>> 
>>> On Jun 20, 2016, at 5:10 AM, Sean Dague >> > wrote:
>>> 
>>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
 
> On Jun 14, 2016, at 3:59 PM, Edward Leafe  > wrote:
> 
> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  > wrote:
> 
>> But, if we add another possible state on the defcore side like
>> conditional pass,
>> warning, yellow, etc. (the name doesn't matter) which is used to
>> indicate that
>> things on product X could only pass when strict validation was
>> disabled (and
>> be clear about where and why) then my concerns would be alleviated.
>> I just do
>> not want this to end up not being visible to end users trying to
>> evaluate
>> interoperability of different clouds using the test results.
> 
> +1
> 
> Don't fail them, but don't cover up their incompatibility, either.
> -- Ed Leafe
 
 That’s not my proposal. My requirement is that vendors who want to do
 this
 state exactly which APIs are sending back additional data, and that this
 information be published.
 
 There are different levels of incompatibility. A response with
 additional data
 that can be safely ignored is different from a changed response that
 would
 cause a client to fail.
>>> 
>>> It's actually not different. It's really not.
>>> 
>>> This idea that it's safe to add response data is based on an assumption
>>> that software versions only move forward. If you have a single deploy of
>>> software, that's fine.
>>> 
>>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>>> wild. Which means if we want to support horizontal transfer between
>>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>>> then try to move to Juno. So anything added from Juno -> Mitaka without
>>> signaling has exactly the same client breaking behavior as removing
>>> attributes.
>>> 
>>> Which is why microversions are needed for attribute adds.
>> 
>> I’d like to note that Nova v2.0 is still a supported API, which
>> as far as I understand allows for additional attributes and
>> extensions. That Tempest doesn’t allow for disabling strict
>> checking when using a v2.0 endpoint is a problem.
>> 
>> The reporting of v2.0 in the Marketplace (which is what we do
>> right now) is also a signal to a user that there may be vendor
>> additions to the API.
>> 
>> DefCore doesn’t disallow the use of a 2.0 endpoint as part
>> of the interoperability standard.
> 
> This is a point of confusion.
> 
> The API definition did not allow that. The implementation of the API
> stack did.

And downstream vendors took advantage of that. We may
not like it, but it’s a reality in the current ecosystem.

> In Liberty the v2.0 API is optionally provided by a different backend
> stack that doesn't support extensions.
> In Mitaka it is default v2.0 API on a non extensions backend
> In Newton the old backend is deleted.
> 
> From Newton forward there is still a v2.0 API, but all the code hooks
> that provided facilities for extensions are gone.

It’s really important that the current documentation reflect the
code and intent of the dev team. As of writing this e-mail, 

"• v2 (SUPPORTED) and v2 extensions (SUPPORTED) (Will
be deprecated in the near future.)”[1]

Even with this being removed in Newton, DefCore still has
to allow for it in every supported version.

-Chris

[1] http://docs.openstack.org/developer/nova/

>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/22/2016 01:59 PM, Chris Hoge wrote:
> 
>> On Jun 20, 2016, at 5:10 AM, Sean Dague > > wrote:
>>
>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>>
 On Jun 14, 2016, at 3:59 PM, Edward Leafe >>> > wrote:

 On Jun 14, 2016, at 5:50 PM, Matthew Treinish >>> > wrote:

> But, if we add another possible state on the defcore side like
> conditional pass,
> warning, yellow, etc. (the name doesn't matter) which is used to
> indicate that
> things on product X could only pass when strict validation was
> disabled (and
> be clear about where and why) then my concerns would be alleviated.
> I just do
> not want this to end up not being visible to end users trying to
> evaluate
> interoperability of different clouds using the test results.

 +1

 Don't fail them, but don't cover up their incompatibility, either.
 -- Ed Leafe
>>>
>>> That’s not my proposal. My requirement is that vendors who want to do
>>> this
>>> state exactly which APIs are sending back additional data, and that this
>>> information be published.
>>>
>>> There are different levels of incompatibility. A response with
>>> additional data
>>> that can be safely ignored is different from a changed response that
>>> would
>>> cause a client to fail.
>>
>> It's actually not different. It's really not.
>>
>> This idea that it's safe to add response data is based on an assumption
>> that software versions only move forward. If you have a single deploy of
>> software, that's fine.
>>
>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>> wild. Which means if we want to support horizontal transfer between
>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>> then try to move to Juno. So anything added from Juno -> Mitaka without
>> signaling has exactly the same client breaking behavior as removing
>> attributes.
>>
>> Which is why microversions are needed for attribute adds.
> 
> I’d like to note that Nova v2.0 is still a supported API, which
> as far as I understand allows for additional attributes and
> extensions. That Tempest doesn’t allow for disabling strict
> checking when using a v2.0 endpoint is a problem.
> 
> The reporting of v2.0 in the Marketplace (which is what we do
> right now) is also a signal to a user that there may be vendor
> additions to the API.
> 
> DefCore doesn’t disallow the use of a 2.0 endpoint as part
> of the interoperability standard.

This is a point of confusion.

The API definition did not allow that. The implementation of the API
stack did.

In Liberty the v2.0 API is optionally provided by a different backend
stack that doesn't support extensions.
In Mitaka it is default v2.0 API on a non extensions backend
In Newton the old backend is deleted.

From Newton forward there is still a v2.0 API, but all the code hooks
that provided facilities for extensions are gone.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 20, 2016, at 5:10 AM, Sean Dague  wrote:
> 
> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>> 
>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
>>> 
>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
>>> 
 But, if we add another possible state on the defcore side like conditional 
 pass,
 warning, yellow, etc. (the name doesn't matter) which is used to indicate 
 that
 things on product X could only pass when strict validation was disabled 
 (and
 be clear about where and why) then my concerns would be alleviated. I just 
 do
 not want this to end up not being visible to end users trying to evaluate
 interoperability of different clouds using the test results.
>>> 
>>> +1
>>> 
>>> Don't fail them, but don't cover up their incompatibility, either.
>>> -- Ed Leafe
>> 
>> That’s not my proposal. My requirement is that vendors who want to do this
>> state exactly which APIs are sending back additional data, and that this
>> information be published.
>> 
>> There are different levels of incompatibility. A response with additional 
>> data
>> that can be safely ignored is different from a changed response that would
>> cause a client to fail.
> 
> It's actually not different. It's really not.
> 
> This idea that it's safe to add response data is based on an assumption
> that software versions only move forward. If you have a single deploy of
> software, that's fine.
> 
> However as noted, we've got production clouds on Juno <-> Mitaka in the
> wild. Which means if we want to support horizontal transfer between
> clouds, the user experienced timeline might be start on a Mitaka cloud,
> then try to move to Juno. So anything added from Juno -> Mitaka without
> signaling has exactly the same client breaking behavior as removing
> attributes.
> 
> Which is why microversions are needed for attribute adds.

I’d like to note that Nova v2.0 is still a supported API, which
as far as I understand allows for additional attributes and
extensions. That Tempest doesn’t allow for disabling strict
checking when using a v2.0 endpoint is a problem.

The reporting of v2.0 in the Marketplace (which is what we do
right now) is also a signal to a user that there may be vendor
additions to the API.

DefCore doesn’t disallow the use of a 2.0 endpoint as part
of the interoperability standard.

-Chris


>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Chris Hoge]

> On Jun 20, 2016, at 6:56 AM, Doug Hellmann  wrote:
> 

> 
> I'm also, I think, edging away from the "we need to find a compromise"
> camp into the "why is this turning into such a big deal" camp. How did
> we get into a situation where the community has set a clear direction,
> the trademark certification system has a long lead time built in, and
> vendors are still not able to maintain certification?

If I had to make an educated guess, it’s because product managers
have to produce a roadmap with goals and features that consider
both what’s happening upstream, what is currently deployed, and
existing customers.

Just pulling attributes that were once ‘ok’ within the ecosystem and
now aren’t (even with lead time) isn’t as easy as “just change the
response”. It takes time, and given the year-long cycle that DefCore
has adopted for re-testing and the relative youth of the OpenStack
Powered Trademark program, it’s not unsurprising that a few
clouds have been hit by this.

I’ve spoken with all three vendors who are being hit by this, and
two definitely have a longer lead time to work on this. The third
is using the extra responses internally and are currently working
on changing their custom tools to get the extra information in
a different way.

I’ve also spoken with another vendor who is going to be caught
by it, though, and have explained the situation to them and
they are considering their options.

I am now actively telling vendors who are still sending additional
properties to start with with the Nova team upstream to have
their additional properties micro-versioned.

> Are the systems
> running modified versions of newer OpenStack that can't be certified
> with older versions of Tempest?

With the volume of bug fixes, refactoring, and general improvements
from the last year, I’d say no.

-Chris

> 
> Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Mark Voelker's message of 2016-06-20 13:24:46 +:
> 
> > On Jun 20, 2016, at 8:46 AM, Doug Hellmann  wrote:
> > 
> > Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
> > 
> >> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> >> 
> >> 
> >>> I don't think DefCore actually needs to change old versions of Tempest,
> >>> but maybe Chris or Mark can verify that?
> >> 
> >> So if I’m groking this correctly, there’s kind of two scenarios
> >> being painted here.  One is the “LCD” approach where we use the
> >> $osversion-eol version of Tempest, where $osversion matches the
> >> oldest version covered in a Guideline.  The other is to use the
> >> start-of-$osversion version of Tempest where $osversion is the
> >> OpenStack version after the most recent one in the Guideline.  The
> >> former may result in some fairly long-lived flags, and the latter
> >> is actually not terribly different than what we do today I think.
> >> Let me try to talk through both...
> >> 
> >> In some cases, tests get flagged in the Guidelines because of
> >> bugs in the test or because the test needs refactoring.  The
> >> underlying Capabilities the those tests are testing actually work
> >> fine.  Once we identify such an issue, the test can be fixed…in
> >> master.  Under the first scenario, this potentially creates some
> >> very long-lived flags:
> >> 
> >> 2016.01 is the most current Guideline right now covers Juno, Kilo,
> >> Liberty (and Mitaka after it was released).  It’s one of the two
> >> Guidelines that you can use if you want an OpenStack Powered license
> >> from he Foundation, $vendor wants to run it against their shiny new
> >> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
> >> they find a test issue, and we flag it.  A few weeks later, a fix
> >> lands in Tempest.  Several months later the next Guideline rolls
> >> around: the oldest covered release is Kilo and we start telling
> >> people to use the Kilo-EOL version of Tempest.  That doesn’t have
> >> the fix, so the flag stays.  Another six months goes by and we get
> >> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
> >> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
> >> that's the first version that includes the fix.
> >> 
> >> Generally speaking long lived flags aren’t so great because it
> >> means the tests are not required…which means there’s less or no
> >> assurance that the capabilities they test for actually work in the
> >> clouds that adhere to those Guidelines.  So, the worst-case scenario
> >> here looks kind of ugly.
> >> 
> >> As Matt correctly pointed out though, the capabilities DefCore
> >> selects for are generally pretty stable API’s that are long-lived
> >> across many releases, so we haven’t run into a lot of issues running
> >> pretty new versions of Tempest against older clouds to date.  In
> >> fact I’m struggling to think of a time we’ve flagged something
> >> because someone complained the test wasn’t runnable against an older
> >> release covered by the Guideline in question.  I can think of plenty
> >> of times where we’ve flagged something due to a test issue though…keep
> >> in mind we’re still in pretty formative times with DefCore here
> >> where these tests are starting to be used in a new way for the first
> >> time.  Anyway, as Matt points out we could potentially use a much
> >> newer Tempest tag: tag=11 (which is the start of Newton development
> >> and is a roughly 2 month old version of Tempest).  Next Guideline
> >> rolls around, we use the tag for start-of-ocata, and we get the fix
> >> and can drop the flag.
> >> 
> >> Today, RefStack client by default checks out a specific SHA of
> >> Tempest [1] (it actually did use a tag at some point in the past,
> >> and still can).  When we see a fix for a flagged test go in, we or
> >> the Refstack folks can do a quick test to make sure everything’s
> >> in order and then update that SHA to match the version with the
> >> fix.  That way we’re relatively sure we have a version that works
> >> today, and will work when we drop the flag in the next Guideline
> >> too.  When we finalize that next Guideline, we also update the
> >> test-repositories section of the new Guideline that Matt pointed
> >> to earlier to reflect the best-known version on the day the Guideline
> >> was sent to the Board for approval.  One added benefit of this
> >> approach is that people running the tests today may get a version
> >> of Tempest that includes a fix for a flagged test.  A flagged test
> >> isn’t required, but it does get run—and now will show a passing
> >> result, so we have data that says “this provider actually does
> >> support this capability (even though it’s flagged), and the test
> >> does indeed seem to be working."
> >> 
> >> 
> >> So, that’s actually not hugely different from the second scenario
> >> I think?  Or did I miss something there?
> > 
> > What I was proposing is tha

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

> On Jun 20, 2016, at 8:46 AM, Doug Hellmann  wrote:
> 
> Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:
> 
>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> 
>> 
>>> I don't think DefCore actually needs to change old versions of Tempest,
>>> but maybe Chris or Mark can verify that?
>> 
>> So if I’m groking this correctly, there’s kind of two scenarios
>> being painted here.  One is the “LCD” approach where we use the
>> $osversion-eol version of Tempest, where $osversion matches the
>> oldest version covered in a Guideline.  The other is to use the
>> start-of-$osversion version of Tempest where $osversion is the
>> OpenStack version after the most recent one in the Guideline.  The
>> former may result in some fairly long-lived flags, and the latter
>> is actually not terribly different than what we do today I think.
>> Let me try to talk through both...
>> 
>> In some cases, tests get flagged in the Guidelines because of
>> bugs in the test or because the test needs refactoring.  The
>> underlying Capabilities the those tests are testing actually work
>> fine.  Once we identify such an issue, the test can be fixed…in
>> master.  Under the first scenario, this potentially creates some
>> very long-lived flags:
>> 
>> 2016.01 is the most current Guideline right now covers Juno, Kilo,
>> Liberty (and Mitaka after it was released).  It’s one of the two
>> Guidelines that you can use if you want an OpenStack Powered license
>> from he Foundation, $vendor wants to run it against their shiny new
>> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
>> they find a test issue, and we flag it.  A few weeks later, a fix
>> lands in Tempest.  Several months later the next Guideline rolls
>> around: the oldest covered release is Kilo and we start telling
>> people to use the Kilo-EOL version of Tempest.  That doesn’t have
>> the fix, so the flag stays.  Another six months goes by and we get
>> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
>> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
>> that's the first version that includes the fix.
>> 
>> Generally speaking long lived flags aren’t so great because it
>> means the tests are not required…which means there’s less or no
>> assurance that the capabilities they test for actually work in the
>> clouds that adhere to those Guidelines.  So, the worst-case scenario
>> here looks kind of ugly.
>> 
>> As Matt correctly pointed out though, the capabilities DefCore
>> selects for are generally pretty stable API’s that are long-lived
>> across many releases, so we haven’t run into a lot of issues running
>> pretty new versions of Tempest against older clouds to date.  In
>> fact I’m struggling to think of a time we’ve flagged something
>> because someone complained the test wasn’t runnable against an older
>> release covered by the Guideline in question.  I can think of plenty
>> of times where we’ve flagged something due to a test issue though…keep
>> in mind we’re still in pretty formative times with DefCore here
>> where these tests are starting to be used in a new way for the first
>> time.  Anyway, as Matt points out we could potentially use a much
>> newer Tempest tag: tag=11 (which is the start of Newton development
>> and is a roughly 2 month old version of Tempest).  Next Guideline
>> rolls around, we use the tag for start-of-ocata, and we get the fix
>> and can drop the flag.
>> 
>> Today, RefStack client by default checks out a specific SHA of
>> Tempest [1] (it actually did use a tag at some point in the past,
>> and still can).  When we see a fix for a flagged test go in, we or
>> the Refstack folks can do a quick test to make sure everything’s
>> in order and then update that SHA to match the version with the
>> fix.  That way we’re relatively sure we have a version that works
>> today, and will work when we drop the flag in the next Guideline
>> too.  When we finalize that next Guideline, we also update the
>> test-repositories section of the new Guideline that Matt pointed
>> to earlier to reflect the best-known version on the day the Guideline
>> was sent to the Board for approval.  One added benefit of this
>> approach is that people running the tests today may get a version
>> of Tempest that includes a fix for a flagged test.  A flagged test
>> isn’t required, but it does get run—and now will show a passing
>> result, so we have data that says “this provider actually does
>> support this capability (even though it’s flagged), and the test
>> does indeed seem to be working."
>> 
>> 
>> So, that’s actually not hugely different from the second scenario
>> I think?  Or did I miss something there?
> 
> What I was proposing is that we keep the certification rules in
> sync with OpenStack versions. If a vendor stays on an old version
> of OpenStack, they can keep using the old (matching) version of
> Tempest to certify.  It's not clear from your description if that's

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Mark Voelker's message of 2016-06-16 20:33:36 +:

> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> 
> 
> > I don't think DefCore actually needs to change old versions of Tempest,
> > but maybe Chris or Mark can verify that?
> 
> So if I’m groking this correctly, there’s kind of two scenarios
> being painted here.  One is the “LCD” approach where we use the
> $osversion-eol version of Tempest, where $osversion matches the
> oldest version covered in a Guideline.  The other is to use the
> start-of-$osversion version of Tempest where $osversion is the
> OpenStack version after the most recent one in the Guideline.  The
> former may result in some fairly long-lived flags, and the latter
> is actually not terribly different than what we do today I think.
> Let me try to talk through both...
> 
> In some cases, tests get flagged in the Guidelines because of
> bugs in the test or because the test needs refactoring.  The
> underlying Capabilities the those tests are testing actually work
> fine.  Once we identify such an issue, the test can be fixed…in
> master.  Under the first scenario, this potentially creates some
> very long-lived flags:
> 
> 2016.01 is the most current Guideline right now covers Juno, Kilo,
> Liberty (and Mitaka after it was released).  It’s one of the two
> Guidelines that you can use if you want an OpenStack Powered license
> from he Foundation, $vendor wants to run it against their shiny new
> Mitaka cloud.  They run the Juno EOL version of Tempest (tag=8),
> they find a test issue, and we flag it.  A few weeks later, a fix
> lands in Tempest.  Several months later the next Guideline rolls
> around: the oldest covered release is Kilo and we start telling
> people to use the Kilo-EOL version of Tempest.  That doesn’t have
> the fix, so the flag stays.  Another six months goes by and we get
> a Guideline and we’re up to the Liberty-EOL version of Tempest.  No
> fix, flag stays.  Six more months, and now we’re at Mitaka-EOL, and
> that's the first version that includes the fix.
>
> Generally speaking long lived flags aren’t so great because it
> means the tests are not required…which means there’s less or no
> assurance that the capabilities they test for actually work in the
> clouds that adhere to those Guidelines.  So, the worst-case scenario
> here looks kind of ugly.
> 
> As Matt correctly pointed out though, the capabilities DefCore
> selects for are generally pretty stable API’s that are long-lived
> across many releases, so we haven’t run into a lot of issues running
> pretty new versions of Tempest against older clouds to date.  In
> fact I’m struggling to think of a time we’ve flagged something
> because someone complained the test wasn’t runnable against an older
> release covered by the Guideline in question.  I can think of plenty
> of times where we’ve flagged something due to a test issue though…keep
> in mind we’re still in pretty formative times with DefCore here
> where these tests are starting to be used in a new way for the first
> time.  Anyway, as Matt points out we could potentially use a much
> newer Tempest tag: tag=11 (which is the start of Newton development
> and is a roughly 2 month old version of Tempest).  Next Guideline
> rolls around, we use the tag for start-of-ocata, and we get the fix
> and can drop the flag.
> 
> Today, RefStack client by default checks out a specific SHA of
> Tempest [1] (it actually did use a tag at some point in the past,
> and still can).  When we see a fix for a flagged test go in, we or
> the Refstack folks can do a quick test to make sure everything’s
> in order and then update that SHA to match the version with the
> fix.  That way we’re relatively sure we have a version that works
> today, and will work when we drop the flag in the next Guideline
> too.  When we finalize that next Guideline, we also update the
> test-repositories section of the new Guideline that Matt pointed
> to earlier to reflect the best-known version on the day the Guideline
> was sent to the Board for approval.  One added benefit of this
> approach is that people running the tests today may get a version
> of Tempest that includes a fix for a flagged test.  A flagged test
> isn’t required, but it does get run—and now will show a passing
> result, so we have data that says “this provider actually does
> support this capability (even though it’s flagged), and the test
> does indeed seem to be working."
> 
> 
> So, that’s actually not hugely different from the second scenario
> I think?  Or did I miss something there?

What I was proposing is that we keep the certification rules in
sync with OpenStack versions. If a vendor stays on an old version
of OpenStack, they can keep using the old (matching) version of
Tempest to certify.  It's not clear from your description if that's
allowed now.

We would want to expose the version of OpenStack, the version of
Tempest, any flagged tests, and any configuration settings that
change 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Sean Dague]

On 06/14/2016 07:19 PM, Chris Hoge wrote:
> 
>> On Jun 14, 2016, at 3:59 PM, Edward Leafe  wrote:
>>
>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish  wrote:
>>
>>> But, if we add another possible state on the defcore side like conditional 
>>> pass,
>>> warning, yellow, etc. (the name doesn't matter) which is used to indicate 
>>> that
>>> things on product X could only pass when strict validation was disabled (and
>>> be clear about where and why) then my concerns would be alleviated. I just 
>>> do
>>> not want this to end up not being visible to end users trying to evaluate
>>> interoperability of different clouds using the test results.
>>
>> +1
>>
>> Don't fail them, but don't cover up their incompatibility, either.
>> -- Ed Leafe
> 
> That’s not my proposal. My requirement is that vendors who want to do this
> state exactly which APIs are sending back additional data, and that this
> information be published.
> 
> There are different levels of incompatibility. A response with additional data
> that can be safely ignored is different from a changed response that would
> cause a client to fail.

It's actually not different. It's really not.

This idea that it's safe to add response data is based on an assumption
that software versions only move forward. If you have a single deploy of
software, that's fine.

However as noted, we've got production clouds on Juno <-> Mitaka in the
wild. Which means if we want to support horizontal transfer between
clouds, the user experienced timeline might be start on a Mitaka cloud,
then try to move to Juno. So anything added from Juno -> Mitaka without
signaling has exactly the same client breaking behavior as removing
attributes.

Which is why microversions are needed for attribute adds.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Ken'ichi Ohmichi]

2016-06-16 2:26 GMT-07:00 Morgan Fainberg :
> On Wed, Jun 15, 2016 at 11:54 PM, Ken'ichi Ohmichi 
> wrote:
>>
>> This discussion was expected when we implemented the Tempest patch,
>> then I sent a mail to defcore comittee[1]
>> As the above ml, "A DefCore Guideline typically covers three OpenStack
>> releases".
>> That means the latest guideline needs to cover Mitaka, Liberty and Kilo,
>> right?
>>
>> In the Kilo development, we(nova team) have already considered
>> additional properties are not good for the interoperability.
>> And the stable_api.rst of [2] which is contained in Kilo says we need
>> to implement new features without extensions.
>> However, there are Kilo+ clouds which are extended with vendors' own
>> extensions, right?
>>
>> My concern of allowing additional properties on interoperability tests is
>> that
>>  - users can move from pure OpenStack clouds to non-pure OpenStack
>> clouds which implement vender specific properties
>>  - but users cannot move from non-pure OpenStack clouds if users
>> depend on the properties
>> even if these clouds are certificated on the same interoperability tests.
>>
>
> The end goal is 100% to get everyone consistent with no "extra" data being
> passed out of the APIs and certified on the same tests.

Yeah, I am appreciated that everyone agree with the non-extra data as
the final goal.

> However, right now we have an issue where vendors/operators are lagging on
> getting this cleaned up. Since this is the first round of certifications
> (among other things), the proposal is to support/manage this in a way that
> gives a bit more of a grace period while the deployers/operators finish
> moving away from custom properties (as i understand it the ones affected
> have communicated that they are working on meeting this goal; Chris, please
> correct me if I am wrong).
>
> Your concerns are spot on, and at the end of this "greylist" window ( at the
> " 2017.01" defcore guideline ), this grace period will expire and everyone
> will be expected to be compatible without the "Extra" data. Part of the
> process of doing these programs is working to refine the process (and
> sometimes make exceptions in the early stages) until the workflow is
> established and understood. It is not expected to continue nor extend the
> period beyond the firm end point Chris highlighted. I would not support this
> proposal if it was open ended.

The greylist seems a good idea, and I am not so strongly against the idea.
However, I have still some questions about this direction.

I am thinking most important API of Nova is "create a server" API for
the interoperability, because most users want to use servers on
OpenStack clouds.
However, I am guessing most venders which cannot be passed through
current strict Tempest are customizing this API.
So if this API on the greylist on most venders' tests, the
interoperability seems a little meaningless.
Is that expected now?

One more question is that how many venders cannot pass through current Tempest?
100%? or 20%?
If 5% venders cannot pass, I guess we can say "the certification is
failed" to the venders.
I'd like to know current situation for expecting our future so that we
will need to mark this "greylist" as deprecated soon and need to know
how progress at some steps/cycles of venders.

Thanks
Ken Omichi

---
>> ---
>> [1]:
>> http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
>> [2]: https://review.openstack.org/#/c/162912
>>
>> 2016-06-14 16:37 GMT-07:00 Chris Hoge :
>> > Top posting one note and direct comments inline, I’m proposing
>> > this as a member of the DefCore working group, but this
>> > proposal itself has not been accepted as the forward course of
>> > action by the working group. These are my own views as the
>> > administrator of the program and not that of the working group
>> > itself, which may independently reject the idea outside of the
>> > response from the upstream devs.
>> >
>> > I posted a link to this thread to the DefCore mailing list to make
>> > that working group aware of the outstanding issues.
>> >
>> > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
>> > wrote:
>> >
>> > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
>> >
>> > Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>> >
>> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>> >
>> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
>> >
>> > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
>> >
>> > Last year, in response to Nova micro-versioning and extension
>> > updates[1],
>> > the QA team added strict API schema checking to Tempest to ensure that
>> > no additional properties were added to Nova API responses[2][3]. In the
>> > last year, at least three vendors participating the the OpenStack
>> > Powered
>> > Trademark program have been impacted by this change, two of which
>> > reported this to the DefCore Working Group mailing list

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matt Riedemann]

On 6/14/2016 6:37 PM, Chris Hoge wrote:

I’m beginning to wonder if we need to make DefCore use release
branches then back-port bug-fixes and relevant features additions
as necessary.



To clarify, do you mean use release branches from the upstream repos? 
Because if DefCore is supporting 2 years of releases, that's going to 
conflict with the 18 month lifespan of branches upstream, i.e. kilo is 
already EOL. So backporting to stable/juno or stable/kilo at this point 
isn't possible, at least in the official openstack repos.


And backporting features to stable branches would break our stable 
branch policy.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matt Riedemann]

On 6/14/2016 6:19 PM, Chris Hoge wrote:

One would hope that micro-versions would be able to address this exact
issue for vendors by giving them a means to propose optional but
well-defined API response additions (not extensions) that are defined
upstream and usable by all vendors. If it’s not too off topic, can someone
from the Nova team explain how something like that would work (if it
would at all)?

-Chris
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



The 2.4 microversion is about as simple as it gets to adding more data 
to a response:


http://docs.openstack.org/developer/nova/api_microversion_history.html#id3

If you're requesting microversion >=2.4 you get the 'reserved' flag back 
when showing details on a fixed IP, else it's omitted from the response 
(as in v2.0 and v2.1).


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mike Perez]

On 23:53 Jun 17, Matthew Treinish wrote:
> On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote:
> > On 15:12 Jun 14, Matthew Treinish wrote:



> > > I don't think backwards compatibility policies really apply to what what 
> > > define
> > > as the set of tests that as a community we are saying a vendor has to 
> > > pass to
> > > say they're OpenStack. From my perspective as a community we either take 
> > > a hard
> > > stance on this and say to be considered an interoperable cloud (and to 
> > > get the
> > > trademark) you have to actually have an interoperable product. We slowly 
> > > ratchet
> > > up the requirements every 6 months, there isn't any implied backwards
> > > compatibility in doing that. You passed in the past but not in the newer 
> > > stricter
> > > guidelines.
> > > 
> > > Also, even if I did think it applied, we're not talking about a change 
> > > which
> > > would fall into breaking that. The change was introduced a year and half 
> > > ago
> > > during kilo and landed a year ago during liberty:
> > > 
> > > https://review.openstack.org/#/c/156130/
> > > 
> > > That's way longer than our normal deprecation period of 3 months and a 
> > > release
> > > boundary.
> > 
> > 
> > 
> > What kind of communication happens today for these changes? There are so 
> > many
> > channels/high volume mailing lists a downstream deployer is expected by the
> > community to listening in. Some disruptive change being introduced a year or
> > longer ago can still be communicated poorly.
> 
> Sure, I agree with that, but I don't think this was necessarily communicated
> poorly. This has been already mentioned a few times on this thread but:
> 
> It was talked about on openstack-dev:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
> 
> On the defcore list: (which is definitely not high volume/traffic ML)
> 
> http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html
> 
> This was also raised as an issue for 1 vendor ~6 months ago. (which is also 
> the
> same duration of the hard deadline being discussed in this thread):
> 
> http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
> 
> IMHO, this was more than enough time to introduce a fix or workaround on their
> end. Likely the easiest being just adding an extra nova-api endpoint with the
> extensions disabled.
> 
> I don't have any links or other evidence to point to, but I know that this
> exact topic has been discussed with with people from the vendors having
> difficulties during sessions at at least one of the 2 summits and/or 2 QA
> midcycle meetups since this change landed. I really don't think this is a
> communication problem or unfair surprise for anyone.
> 
> There might be more too, but I don't remember every conversation that I've had
> in the community over the past year. (or where to find the links to point to)

Thanks for the references. So these references show:

* DefCore was aware of these changes long ago.
* Vendors were aware of these changes long ago.
* Referenced vendor is still failing after knowing about this change for six
  months.

Question to DefCore, what were you doing in this time frame to prepare vendors
for success with this change rolling out in order to keep a healthy market
place?

> > Just like we've done with Reno in communicating better about disruptive 
> > changes
> > in release notes, what tells teams like DefCore about changes with Tempest?
> > (I looked in release.o.o for tempest release notes, although maybe I missed
> > it?)
> 
> Yes, tempest has release notes, they are here:
> 
> http://docs.openstack.org/releasenotes/tempest/
> 
> But, the change in question predates the existence of reno and centralized
> release notes for everything in openstack.
> 
> If this change were pushed today it would definitely be included in the 
> release
> notes. We also would do the same things, put it on the dev list, put it on the
> defcore list. (although probably as a standalone thread this time) I also 
> think
> we'd probably ping hogepodge on irc about it too just so he could also raise 
> it
> up on the defcore side. (which we might have done back then too) Defcore and
> tempest are tightly coupled so we do have pretty constant communication around
> changes being made. But, I do admit we have better mechanisms in place today
> to communicate this kind of change, and hopefully this would be handled better
> now.

This is great! I hope people who have use cases with Tempest are using these
release notes for future big changes.

> > 
> > Since some members of DefCore have interest in making the market place 
> > healthy,
> > what is DefCore doing today to communicate these disruptive changes early to
> > deployers? Did it not happen in this particular case because:
> > 
> > * DefCore has no one working closely in the Tempest project to flag things?
> > * Defcore does work closely with Tempest, but somehow the communication for
> 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Fri, Jun 17, 2016 at 04:26:49PM -0700, Mike Perez wrote:
> On 15:12 Jun 14, Matthew Treinish wrote:
> > On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> 
> 
> 
> > > We have basically three options.
> > > 
> > > 1. Tell deployers who are trying to do the right for their immediate
> > >users that they can't use the trademark.
> > > 
> > > 2. Flag the related tests or remove them from the DefCore enforcement
> > >suite entirely.
> > > 
> > > 3. Be flexible about giving consumers of Tempest time to meet the
> > >new requirement by providing a way to disable the checks.
> > > 
> > > Option 1 goes against our own backwards compatibility policies.
> > 
> > I don't think backwards compatibility policies really apply to what what 
> > define
> > as the set of tests that as a community we are saying a vendor has to pass 
> > to
> > say they're OpenStack. From my perspective as a community we either take a 
> > hard
> > stance on this and say to be considered an interoperable cloud (and to get 
> > the
> > trademark) you have to actually have an interoperable product. We slowly 
> > ratchet
> > up the requirements every 6 months, there isn't any implied backwards
> > compatibility in doing that. You passed in the past but not in the newer 
> > stricter
> > guidelines.
> > 
> > Also, even if I did think it applied, we're not talking about a change which
> > would fall into breaking that. The change was introduced a year and half ago
> > during kilo and landed a year ago during liberty:
> > 
> > https://review.openstack.org/#/c/156130/
> > 
> > That's way longer than our normal deprecation period of 3 months and a 
> > release
> > boundary.
> 
> 
> 
> What kind of communication happens today for these changes? There are so many
> channels/high volume mailing lists a downstream deployer is expected by the
> community to listening in. Some disruptive change being introduced a year or
> longer ago can still be communicated poorly.

Sure, I agree with that, but I don't think this was necessarily communicated
poorly. This has been already mentioned a few times on this thread but:

It was talked about on openstack-dev:

http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html

On the defcore list: (which is definitely not high volume/traffic ML)

http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html

This was also raised as an issue for 1 vendor ~6 months ago. (which is also the
same duration of the hard deadline being discussed in this thread):

http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html

IMHO, this was more than enough time to introduce a fix or workaround on their
end. Likely the easiest being just adding an extra nova-api endpoint with the
extensions disabled.

I don't have any links or other evidence to point to, but I know that this
exact topic has been discussed with with people from the vendors having
difficulties during sessions at at least one of the 2 summits and/or 2 QA
midcycle meetups since this change landed. I really don't think this is a
communication problem or unfair surprise for anyone.

There might be more too, but I don't remember every conversation that I've had
in the community over the past year. (or where to find the links to point to)

> 
> Just like we've done with Reno in communicating better about disruptive 
> changes
> in release notes, what tells teams like DefCore about changes with Tempest?
> (I looked in release.o.o for tempest release notes, although maybe I missed
> it?)

Yes, tempest has release notes, they are here:

http://docs.openstack.org/releasenotes/tempest/

But, the change in question predates the existence of reno and centralized
release notes for everything in openstack.

If this change were pushed today it would definitely be included in the release
notes. We also would do the same things, put it on the dev list, put it on the
defcore list. (although probably as a standalone thread this time) I also think
we'd probably ping hogepodge on irc about it too just so he could also raise it
up on the defcore side. (which we might have done back then too) Defcore and
tempest are tightly coupled so we do have pretty constant communication around
changes being made. But, I do admit we have better mechanisms in place today
to communicate this kind of change, and hopefully this would be handled better
now.

> 
> Since some members of DefCore have interest in making the market place 
> healthy,
> what is DefCore doing today to communicate these disruptive changes early to
> deployers? Did it not happen in this particular case because:
> 
> * DefCore has no one working closely in the Tempest project to flag things?
> * Defcore does work closely with Tempest, but somehow the communication for
>   this was missed?
> * Not having clear dep

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mike Perez]

On 15:12 Jun 14, Matthew Treinish wrote:
> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:



> > We have basically three options.
> > 
> > 1. Tell deployers who are trying to do the right for their immediate
> >users that they can't use the trademark.
> > 
> > 2. Flag the related tests or remove them from the DefCore enforcement
> >suite entirely.
> > 
> > 3. Be flexible about giving consumers of Tempest time to meet the
> >new requirement by providing a way to disable the checks.
> > 
> > Option 1 goes against our own backwards compatibility policies.
> 
> I don't think backwards compatibility policies really apply to what what 
> define
> as the set of tests that as a community we are saying a vendor has to pass to
> say they're OpenStack. From my perspective as a community we either take a 
> hard
> stance on this and say to be considered an interoperable cloud (and to get the
> trademark) you have to actually have an interoperable product. We slowly 
> ratchet
> up the requirements every 6 months, there isn't any implied backwards
> compatibility in doing that. You passed in the past but not in the newer 
> stricter
> guidelines.
> 
> Also, even if I did think it applied, we're not talking about a change which
> would fall into breaking that. The change was introduced a year and half ago
> during kilo and landed a year ago during liberty:
> 
> https://review.openstack.org/#/c/156130/
> 
> That's way longer than our normal deprecation period of 3 months and a release
> boundary.



What kind of communication happens today for these changes? There are so many
channels/high volume mailing lists a downstream deployer is expected by the
community to listening in. Some disruptive change being introduced a year or
longer ago can still be communicated poorly.

Just like we've done with Reno in communicating better about disruptive changes
in release notes, what tells teams like DefCore about changes with Tempest?
(I looked in release.o.o for tempest release notes, although maybe I missed
it?)

Since some members of DefCore have interest in making the market place healthy,
what is DefCore doing today to communicate these disruptive changes early to
deployers? Did it not happen in this particular case because:

* DefCore has no one working closely in the Tempest project to flag things?
* Defcore does work closely with Tempest, but somehow the communication for
  this was missed?
* Not having clear deprecation notices because release notes in the Tempest
  don't exist (see above)?

This all just sounds like a communication problem, and it makes me sad to
interpret this thread as people being angry with deployers as a result. How
about we not think the worse of people that are trying to prove our project
being successful and start working with them?

With that said I agree with this strict checking in tests. Deployments need to
stop defining the community defined APIs.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[John Garbutt]

On 15 June 2016 at 02:37, Sean Dague  wrote:
> On 06/14/2016 07:28 PM, Monty Taylor wrote:
>>
>> On 06/14/2016 05:42 PM, Doug Hellmann wrote:
>
> 
>>
>> I think this is the most important thing to me as it relates to this.
>> I'm obviously a huge proponent of clouds behaving more samely. But I
>> also think that, as Doug nicely describes above, we've sort of backed in
>> to removing something without a deprecation window ... largely because
>> of the complexities involved with the system here - and I'd like to make
>> sure that when we are being clear about behavior changes that we give
>> the warning period so that people can adapt.

+1

> I also think that "pass" to "pass *"  is useful social incentive. While I
> think communication of this new direction has happened pretty broadly,
> organizations are complex places, and it didn't filter everywhere it needed
> to with the urgency that was probably needed.
>
> "pass *"  * - with a greylist which goes away in 6 months
>
> Will hopefully be a reasonable enough push to get the behavior we want,
> which is everyone publishing the same interface.

+1

I know I am going back in time, but +1 all the same.

I like how this pushes forward the of interoperability cause.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Mark Voelker]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



On Jun 16, 2016, at 2:25 PM, Matthew Treinish  wrote:

On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
Top posting one note and direct comments inline, I’m proposing
this as a member of the DefCore working group, but this
proposal itself has not been accepted as the forward course of
action by the working group. These are my own views as the
administrator of the program and not that of the working group
itself, which may independently reject the idea outside of the
response from the upstream devs.

I posted a link to this thread to the DefCore mailing list to make
that working group aware of the outstanding issues.

On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:

On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
Last year, in response to Nova micro-versioning and extension updates[1],
the QA team added strict API schema checking to Tempest to ensure that
no additional properties were added to Nova API responses[2][3]. In the
last year, at least three vendors participating the the OpenStack Powered
Trademark program have been impacted by this change, two of which
reported this to the DefCore Working Group mailing list earlier this year[4].

The DefCore Working Group determines guidelines for the OpenStack Powered
program, which includes capabilities with associated functional tests
from Tempest that must be passed, and designated sections with associated
upstream code [5][6]. In determining these guidelines, the working group
attempts to balance the future direction of development with lagging
indicators of deployments and user adoption.

After a tremendous amount of consideration, I believe that the DefCore
Working Group needs to implement a temporary waiver for the strict API
checking requirements that were introduced last year, to give downstream
deployers more time to catch up with the strict micro-versioning
requirements determined by the Nova/Compute team and enforced by the
Tempest/QA team.

I'm very much opposed to this being done. If we're actually concerned with
interoperability and verify that things behave in the same manner between 
multiple
clouds then doing this would be a big step backwards. The fundamental disconnect
here is that the vendors who have implemented out of band extensions or were
taking advantage of previously available places to inject extra attributes
believe that doing so means they're interoperable, which is quite far from
reality. **The API is not a place for vendor differentiation.**

This is a temporary measure to address the fact that a large number
of existing tests changed their behavior, rather than having new
tests added to enforce this new requirement. The result is deployments
that previously passed these tests may no longer pass, and in fact
we have several cases where that's true with deployers who are
trying to maintain their own standard of backwards-compatibility
for their end users.

That's not what happened though. The API hasn't changed and the tests haven't
really changed either. We made our enforcement on Nova's APIs a bit stricter to
ensure nothing unexpected appeared. For the most these tests work on any version
of OpenStack. (we only test it in the gate on supported stable releases, but I
don't expect things to have drastically shifted on older releases) It also
doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the
only case it ever fails is when you run something extra, not from the community,
either as an extension (which themselves are going away [1]) or another service
that wraps nova or imitates nova. I'm personally not comfortable saying those
extras are ever part of the OpenStack APIs.

We have basically three options.

1. Tell deployers who are trying to do the right for their immediate
 users that they can't use the trademark.

2. Flag the related tests or remove them from the DefCore enforcement
 suite entirely.

3. Be flexible about giving consumers of Tempest time to meet the
 new requirement by providing a way to disable the checks.

Option 1 goes against our own backwards compatibility policies.

I don't think backwards compatibility policies really apply to what what define
as the set of tests that as a community we are saying a vendor has to pass to
say they're OpenStack. From my perspective as a community we either take a hard
stance on thi

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Thu, Jun 16, 2016 at 02:15:47PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
> > On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> > > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > > > Top posting one note and direct comments inline, I’m proposing
> > > > > > this as a member of the DefCore working group, but this
> > > > > > proposal itself has not been accepted as the forward course of
> > > > > > action by the working group. These are my own views as the
> > > > > > administrator of the program and not that of the working group
> > > > > > itself, which may independently reject the idea outside of the
> > > > > > response from the upstream devs.
> > > > > > 
> > > > > > I posted a link to this thread to the DefCore mailing list to make
> > > > > > that working group aware of the outstanding issues.
> > > > > > 
> > > > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
> > > > > > >  wrote:
> > > > > > > 
> > > > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > > > >> -0400:
> > > > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > > > >  Excerpts from Matthew Treinish's message of 2016-06-14 
> > > > > >  14:21:27 -0400:
> > > > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > > >> Last year, in response to Nova micro-versioning and 
> > > > > > >> extension updates[1],
> > > > > > >> the QA team added strict API schema checking to Tempest to 
> > > > > > >> ensure that
> > > > > > >> no additional properties were added to Nova API 
> > > > > > >> responses[2][3]. In the
> > > > > > >> last year, at least three vendors participating the the 
> > > > > > >> OpenStack Powered
> > > > > > >> Trademark program have been impacted by this change, two of 
> > > > > > >> which
> > > > > > >> reported this to the DefCore Working Group mailing list 
> > > > > > >> earlier this year[4].
> > > > > > >> 
> > > > > > >> The DefCore Working Group determines guidelines for the 
> > > > > > >> OpenStack Powered
> > > > > > >> program, which includes capabilities with associated 
> > > > > > >> functional tests
> > > > > > >> from Tempest that must be passed, and designated sections 
> > > > > > >> with associated
> > > > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > > > >> working group
> > > > > > >> attempts to balance the future direction of development with 
> > > > > > >> lagging
> > > > > > >> indicators of deployments and user adoption.
> > > > > > >> 
> > > > > > >> After a tremendous amount of consideration, I believe that 
> > > > > > >> the DefCore
> > > > > > >> Working Group needs to implement a temporary waiver for the 
> > > > > > >> strict API
> > > > > > >> checking requirements that were introduced last year, to 
> > > > > > >> give downstream
> > > > > > >> deployers more time to catch up with the strict 
> > > > > > >> micro-versioning
> > > > > > >> requirements determined by the Nova/Compute team and 
> > > > > > >> enforced by the
> > > > > > >> Tempest/QA team.
> > > > > > > 
> > > > > > > I'm very much opposed to this being done. If we're actually 
> > > > > > > concerned with
> > > > > > > interoperability and verify that things behave in the same 
> > > > > > > manner between multiple
> > > > > > > clouds then doing this would be a big step backwards. The 
> > > > > > > fundamental disconnect
> > > > > > > here is that the vendors who have implemented out of band 
> > > > > > > extensions or were
> > > > > > > taking advantage of previously available places to inject 
> > > > > > > extra attributes
> > > > > > > believe that doing so means they're interoperable, which is 
> > > > > > > quite far from
> > > > > > > reality. **The API is not a place for vendor 
> > > > > > > differentiation.**
> > > > > >  
> > > > > >  This is a temporary measure to address the fact that a large 
> > > > > >  number
> > > > > >  of existing tests changed their behavior, rather than having 
> > > > > >  new
> > > > > >  tests added to enforce this new requirement. The result is 
> > > > > >  deployments
> > > > > >  that previously passed these tests may no longer pass, and in 
> > > > > >  fact
> > > > > >  we have several cases where that's true with deployers who are
> > > > > >  trying to maintain their own standard of 
> > > > > >  backwards-compatibility
> > > > > >  for their end users.
> > > > > > >>> 

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-16 13:56:31 -0400:
> On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> > Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > > Top posting one note and direct comments inline, I’m proposing
> > > > > this as a member of the DefCore working group, but this
> > > > > proposal itself has not been accepted as the forward course of
> > > > > action by the working group. These are my own views as the
> > > > > administrator of the program and not that of the working group
> > > > > itself, which may independently reject the idea outside of the
> > > > > response from the upstream devs.
> > > > > 
> > > > > I posted a link to this thread to the DefCore mailing list to make
> > > > > that working group aware of the outstanding issues.
> > > > > 
> > > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish 
> > > > > >  wrote:
> > > > > > 
> > > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > > >> -0400:
> > > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > > > >  -0400:
> > > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > > >> Last year, in response to Nova micro-versioning and extension 
> > > > > >> updates[1],
> > > > > >> the QA team added strict API schema checking to Tempest to 
> > > > > >> ensure that
> > > > > >> no additional properties were added to Nova API 
> > > > > >> responses[2][3]. In the
> > > > > >> last year, at least three vendors participating the the 
> > > > > >> OpenStack Powered
> > > > > >> Trademark program have been impacted by this change, two of 
> > > > > >> which
> > > > > >> reported this to the DefCore Working Group mailing list 
> > > > > >> earlier this year[4].
> > > > > >> 
> > > > > >> The DefCore Working Group determines guidelines for the 
> > > > > >> OpenStack Powered
> > > > > >> program, which includes capabilities with associated 
> > > > > >> functional tests
> > > > > >> from Tempest that must be passed, and designated sections with 
> > > > > >> associated
> > > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > > >> working group
> > > > > >> attempts to balance the future direction of development with 
> > > > > >> lagging
> > > > > >> indicators of deployments and user adoption.
> > > > > >> 
> > > > > >> After a tremendous amount of consideration, I believe that the 
> > > > > >> DefCore
> > > > > >> Working Group needs to implement a temporary waiver for the 
> > > > > >> strict API
> > > > > >> checking requirements that were introduced last year, to give 
> > > > > >> downstream
> > > > > >> deployers more time to catch up with the strict 
> > > > > >> micro-versioning
> > > > > >> requirements determined by the Nova/Compute team and enforced 
> > > > > >> by the
> > > > > >> Tempest/QA team.
> > > > > > 
> > > > > > I'm very much opposed to this being done. If we're actually 
> > > > > > concerned with
> > > > > > interoperability and verify that things behave in the same 
> > > > > > manner between multiple
> > > > > > clouds then doing this would be a big step backwards. The 
> > > > > > fundamental disconnect
> > > > > > here is that the vendors who have implemented out of band 
> > > > > > extensions or were
> > > > > > taking advantage of previously available places to inject extra 
> > > > > > attributes
> > > > > > believe that doing so means they're interoperable, which is 
> > > > > > quite far from
> > > > > > reality. **The API is not a place for vendor differentiation.**
> > > > >  
> > > > >  This is a temporary measure to address the fact that a large 
> > > > >  number
> > > > >  of existing tests changed their behavior, rather than having new
> > > > >  tests added to enforce this new requirement. The result is 
> > > > >  deployments
> > > > >  that previously passed these tests may no longer pass, and in 
> > > > >  fact
> > > > >  we have several cases where that's true with deployers who are
> > > > >  trying to maintain their own standard of backwards-compatibility
> > > > >  for their end users.
> > > > > >>> 
> > > > > >>> That's not what happened though. The API hasn't changed and the 
> > > > > >>> tests haven't
> > > > > >>> really changed either. We made our enforcement on Nova's APIs a 
> > > > > >>> bit stricter to
> > > > > >>> ensure nothing unexpected appeared. For the most these tests wor

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Matthew Treinish]

On Thu, Jun 16, 2016 at 12:59:41PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > > Top posting one note and direct comments inline, I’m proposing
> > > > this as a member of the DefCore working group, but this
> > > > proposal itself has not been accepted as the forward course of
> > > > action by the working group. These are my own views as the
> > > > administrator of the program and not that of the working group
> > > > itself, which may independently reject the idea outside of the
> > > > response from the upstream devs.
> > > > 
> > > > I posted a link to this thread to the DefCore mailing list to make
> > > > that working group aware of the outstanding issues.
> > > > 
> > > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  
> > > > > wrote:
> > > > > 
> > > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 
> > > > >> -0400:
> > > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > > >  -0400:
> > > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > > >> Last year, in response to Nova micro-versioning and extension 
> > > > >> updates[1],
> > > > >> the QA team added strict API schema checking to Tempest to 
> > > > >> ensure that
> > > > >> no additional properties were added to Nova API responses[2][3]. 
> > > > >> In the
> > > > >> last year, at least three vendors participating the the 
> > > > >> OpenStack Powered
> > > > >> Trademark program have been impacted by this change, two of which
> > > > >> reported this to the DefCore Working Group mailing list earlier 
> > > > >> this year[4].
> > > > >> 
> > > > >> The DefCore Working Group determines guidelines for the 
> > > > >> OpenStack Powered
> > > > >> program, which includes capabilities with associated functional 
> > > > >> tests
> > > > >> from Tempest that must be passed, and designated sections with 
> > > > >> associated
> > > > >> upstream code [5][6]. In determining these guidelines, the 
> > > > >> working group
> > > > >> attempts to balance the future direction of development with 
> > > > >> lagging
> > > > >> indicators of deployments and user adoption.
> > > > >> 
> > > > >> After a tremendous amount of consideration, I believe that the 
> > > > >> DefCore
> > > > >> Working Group needs to implement a temporary waiver for the 
> > > > >> strict API
> > > > >> checking requirements that were introduced last year, to give 
> > > > >> downstream
> > > > >> deployers more time to catch up with the strict micro-versioning
> > > > >> requirements determined by the Nova/Compute team and enforced by 
> > > > >> the
> > > > >> Tempest/QA team.
> > > > > 
> > > > > I'm very much opposed to this being done. If we're actually 
> > > > > concerned with
> > > > > interoperability and verify that things behave in the same manner 
> > > > > between multiple
> > > > > clouds then doing this would be a big step backwards. The 
> > > > > fundamental disconnect
> > > > > here is that the vendors who have implemented out of band 
> > > > > extensions or were
> > > > > taking advantage of previously available places to inject extra 
> > > > > attributes
> > > > > believe that doing so means they're interoperable, which is quite 
> > > > > far from
> > > > > reality. **The API is not a place for vendor differentiation.**
> > > >  
> > > >  This is a temporary measure to address the fact that a large number
> > > >  of existing tests changed their behavior, rather than having new
> > > >  tests added to enforce this new requirement. The result is 
> > > >  deployments
> > > >  that previously passed these tests may no longer pass, and in fact
> > > >  we have several cases where that's true with deployers who are
> > > >  trying to maintain their own standard of backwards-compatibility
> > > >  for their end users.
> > > > >>> 
> > > > >>> That's not what happened though. The API hasn't changed and the 
> > > > >>> tests haven't
> > > > >>> really changed either. We made our enforcement on Nova's APIs a bit 
> > > > >>> stricter to
> > > > >>> ensure nothing unexpected appeared. For the most these tests work 
> > > > >>> on any version
> > > > >>> of OpenStack. (we only test it in the gate on supported stable 
> > > > >>> releases, but I
> > > > >>> don't expect things to have drastically shifted on older releases) 
> > > > >>> It also
> > > > >>> doesn't matter which version of the API you run, v2.0 or v2.1. 
> > >

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2016-06-15 19:27:13 -0400:
> On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
> > Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
> > > Top posting one note and direct comments inline, I’m proposing
> > > this as a member of the DefCore working group, but this
> > > proposal itself has not been accepted as the forward course of
> > > action by the working group. These are my own views as the
> > > administrator of the program and not that of the working group
> > > itself, which may independently reject the idea outside of the
> > > response from the upstream devs.
> > > 
> > > I posted a link to this thread to the DefCore mailing list to make
> > > that working group aware of the outstanding issues.
> > > 
> > > > On Jun 14, 2016, at 3:50 PM, Matthew Treinish  
> > > > wrote:
> > > > 
> > > > On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> > > >> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
> > > >>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
> > >  Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 
> > >  -0400:
> > > > On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > > >> Last year, in response to Nova micro-versioning and extension 
> > > >> updates[1],
> > > >> the QA team added strict API schema checking to Tempest to ensure 
> > > >> that
> > > >> no additional properties were added to Nova API responses[2][3]. 
> > > >> In the
> > > >> last year, at least three vendors participating the the OpenStack 
> > > >> Powered
> > > >> Trademark program have been impacted by this change, two of which
> > > >> reported this to the DefCore Working Group mailing list earlier 
> > > >> this year[4].
> > > >> 
> > > >> The DefCore Working Group determines guidelines for the OpenStack 
> > > >> Powered
> > > >> program, which includes capabilities with associated functional 
> > > >> tests
> > > >> from Tempest that must be passed, and designated sections with 
> > > >> associated
> > > >> upstream code [5][6]. In determining these guidelines, the working 
> > > >> group
> > > >> attempts to balance the future direction of development with 
> > > >> lagging
> > > >> indicators of deployments and user adoption.
> > > >> 
> > > >> After a tremendous amount of consideration, I believe that the 
> > > >> DefCore
> > > >> Working Group needs to implement a temporary waiver for the strict 
> > > >> API
> > > >> checking requirements that were introduced last year, to give 
> > > >> downstream
> > > >> deployers more time to catch up with the strict micro-versioning
> > > >> requirements determined by the Nova/Compute team and enforced by 
> > > >> the
> > > >> Tempest/QA team.
> > > > 
> > > > I'm very much opposed to this being done. If we're actually 
> > > > concerned with
> > > > interoperability and verify that things behave in the same manner 
> > > > between multiple
> > > > clouds then doing this would be a big step backwards. The 
> > > > fundamental disconnect
> > > > here is that the vendors who have implemented out of band 
> > > > extensions or were
> > > > taking advantage of previously available places to inject extra 
> > > > attributes
> > > > believe that doing so means they're interoperable, which is quite 
> > > > far from
> > > > reality. **The API is not a place for vendor differentiation.**
> > >  
> > >  This is a temporary measure to address the fact that a large number
> > >  of existing tests changed their behavior, rather than having new
> > >  tests added to enforce this new requirement. The result is 
> > >  deployments
> > >  that previously passed these tests may no longer pass, and in fact
> > >  we have several cases where that's true with deployers who are
> > >  trying to maintain their own standard of backwards-compatibility
> > >  for their end users.
> > > >>> 
> > > >>> That's not what happened though. The API hasn't changed and the tests 
> > > >>> haven't
> > > >>> really changed either. We made our enforcement on Nova's APIs a bit 
> > > >>> stricter to
> > > >>> ensure nothing unexpected appeared. For the most these tests work on 
> > > >>> any version
> > > >>> of OpenStack. (we only test it in the gate on supported stable 
> > > >>> releases, but I
> > > >>> don't expect things to have drastically shifted on older releases) It 
> > > >>> also
> > > >>> doesn't matter which version of the API you run, v2.0 or v2.1. 
> > > >>> Literally, the
> > > >>> only case it ever fails is when you run something extra, not from the 
> > > >>> community,
> > > >>> either as an extension (which themselves are going away [1]) or 
> > > >>> another service
> > > >>> that wraps nova or imitate

Re: [openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

[Hayes, Graham]

On 16/06/2016 00:30, Matthew Treinish wrote:
> On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote:
>> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700:
>>> Top posting one note and direct comments inline, I’m proposing
>>> this as a member of the DefCore working group, but this
>>> proposal itself has not been accepted as the forward course of
>>> action by the working group. These are my own views as the
>>> administrator of the program and not that of the working group
>>> itself, which may independently reject the idea outside of the
>>> response from the upstream devs.
>>>
>>> I posted a link to this thread to the DefCore mailing list to make
>>> that working group aware of the outstanding issues.
>>>
 On Jun 14, 2016, at 3:50 PM, Matthew Treinish  wrote:

 On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:
> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:
>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:
>>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
 On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:



>>> The current active guidelines cover icehouse through mitaka. The release
>>> of 2016.08 will change that to cover juno through mitaka (with newton
>>> as an add-on to 2016.08 when it’s released). There’s overlap between
>>> the guidelines, so 2016.01 covers juno through mitaka while 2016.08
>>> will cover kilo through newton. Essentially two years of releases.
>>>
> We may also need to consider that test implementation details may
> change, and have a review process within DefCore to help expose
> those changes to make them clearer to deployers.
>
> Fixing the process issue may also mean changing the way we implement
> things in Tempest. In this case, adding a flag helps move ahead
> more smoothly. Perhaps we adopt that as a general policy in the
> future when we make underlying behavioral changes like this to
> existing tests.  Perhaps instead we have a policy that we do not
> change the behavior of existing tests in such significant ways, at
> least if they're tagged as being used by DefCore. I don't know --
> those are things we need to discuss.

 Sure I agree, this thread raises larger issues which need to be figured 
 out.
 But, that is probably an independent discussion.
>>>
>>> I’m beginning to wonder if we need to make DefCore use release
>>> branches then back-port bug-fixes and relevant features additions
>>> as necessary.

What I suggested when the TC decided on keeping the tests in tempest
was to keep them as a tempest plugin.

This would allow the tests to progress overtime, and be versioned,
so that def-core 201x.y would be equal to def-core-tempest-plugin
version 201x.y .

This allows the project developers to continue to develop tests that
match their vision, without causing unforeseen breakages in def-core.

It also allows the def-core tests to target a wider range - tests
that are run currently have no guarantee that they will run against
Kilo, but the def-core plugin could be tested against Kilo known good
clouds in its gate.

Is there any major blocker for moving them?

>> We should definitely have that conversation, to understand what
>> effect it would have both on Tempest and on DefCore.
>>
>
> While from a quick glance this would seem like it would solve some of the
> problems when you start to dig into it you'll see that it actually wouldn't,
> and would just end up causing more issues in the long run. Branchless tempest
> was originally started back at the icehouse release and was implemented to
> actually enforce the API is the same across release boundaries. We had hit 
> many
> issues where incompatibilities inadvertently were introduced into projects' 
> APIs
> which weren't caught because of divergence between tempest master and tempest
> stable/*. If you decide to branch you'll end up having to do the same thing or
> you'll risk the same types of regressions. Testing stable branches against
> master puts you in a weird place where upstream changes could break things for
> your branch and you'd never know until you tried to land something. From a
> tempest dev standpoint branching quite frankly doesn't make any sense.
>
> Also, the other thing to consider is that there wouldn't actually be anything 
> to
> branch on. Tempest always supports whatever releases the community is
> supporting. Every commit is tested against all the branches of OpenStack that
> still exist. When we EOL a stable branch there is no longer anything to run
> tests against. Assuming you're primarily motivated by the fact defcore 
> attempts
> to support branches that no longer have upstream support you wouldn't actually
> be able to do this by branching. When a branch is removed there isn't anything
> for you to test tempest changes against, and merging code without testing it
> first is 

Re: [openstack-dev] [tempest][SR-IOV] tempest breaks Mellanox CI

[Lenny Verkhovsky]

Thanks,
We are investigating the failure.

From: Andrea Frittoli [mailto:andrea.fritt...@gmail.com]
Sent: Thursday, June 16, 2016 2:02 PM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [tempest][SR-IOV] tempest breaks Mellanox CI


On Thu, Jun 16, 2016 at 11:16 AM Moshe Levi 
mailto:mosh...@mellanox.com>> wrote:

Hi all,



A recent change  [1]  in tempest broke all Mellanox CIs.

This is the second time it happened.

After the first time it happened we decided that  Mellanox CI  will comment on 
tempest.

On this time I saw that Mellanox CI was commenting on that  patch with a 
failure but was still got approved - [2] Enabling Mellanox CI as commenting on  
tempest requires  us physical resources  such as servers/NICS because it tests 
SR-IOV.

So I am wandering  what can be done in the future to prevent this from happen 
again.
A message to the DL is a good way to raise awareness on this.
Ensuring the job stability is also important, too many failures will decrease 
the attention on it - that said I don't have numbers for Mellanox CI failure 
rates, so it may be that the pass rate is already good enough.



Anyway we proposed the following fix to tempest [3]
Thanks for fixing this, and sorry about the inconvenience.
Even with the fix in, the Mellanox CI job failed, which if unfortunate.
I issued a re-check.

Andrea Frittoli





[1] - https://review.openstack.org/#/c/320495/

[2] - 
http://13.69.151.247/95/320495/20/check-sriov-tempest/Tempest-Sriov/825304c/testr_results.html.gz

[3] - https://review.openstack.org/#/c/330331/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][SR-IOV] tempest breaks Mellanox CI

[Andrea Frittoli]

On Thu, Jun 16, 2016 at 11:16 AM Moshe Levi  wrote:

> Hi all,
>
>
>
> A recent change  [1]  in tempest broke all Mellanox CIs.
>
> This is the second time it happened.
>
> After the first time it happened we decided that  Mellanox CI  will
> comment on tempest.
>
> On this time I saw that Mellanox CI was commenting on that  patch with a
> failure but was still got approved - [2] Enabling Mellanox CI as commenting
> on  tempest requires  us physical resources  such as servers/NICS because
> it tests SR-IOV.
>
> So I am wandering  what can be done in the future to prevent this from
> happen again.
>
A message to the DL is a good way to raise awareness on this.
Ensuring the job stability is also important, too many failures will
decrease the attention on it - that said I don't have numbers for Mellanox
CI failure rates, so it may be that the pass rate is already good enough.

>
>
> Anyway we proposed the following fix to tempest [3]
>
> Thanks for fixing this, and sorry about the inconvenience.
Even with the fix in, the Mellanox CI job failed, which if unfortunate.
I issued a re-check.

Andrea Frittoli

>
>
>
>
> [1] - https://review.openstack.org/#/c/320495/
>
> [2] -
> http://13.69.151.247/95/320495/20/check-sriov-tempest/Tempest-Sriov/825304c/testr_results.html.gz
>
> [3] - https://review.openstack.org/#/c/330331/
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev