Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Flavio Percoco]

On 17/12/15 02:00 +, Egle Sigler wrote:

Thank you Flavio for bringing this up! We are using tempest tests for
DefCore testing, and we would like to work with anyone willing to increase
coverages in any of the current covered capabilities. We would also like
to hear from the teams when they are planning on removing, changing, or
renaming tests, as that could affect what DefCore tests.

Upcoming DefCore guidelines and tests:
https://github.com/openstack/defcore/blob/master/2016.01.json


+1

It's taking me a bit longer than I expeced but I'm working on a list
of tests that would be great to have in tempest and some that would
perhaps be better to remove.

Thanks for the feedback,
Flavio



Thank you,
Egle

On 12/8/15, 1:25 PM, "Flavio Percoco"  wrote:


Greetings,

I just reviewed a patch in tempest that proposed adding new tests for
the Glance's task API. While I believe it's awesome that folks in the
tempest team keep adding missing tests for projects, I think it'd be
better for the tempest team, the project's team and defcore if we'd
discuss these tests before they are worked on. This should help people
avoid wasting time.

I believe these cases are rare but the benefits of discussing missing
tests across teams could also help prioritizing the work based on what
the teams goals are, what the defcore team needs are, etc.

So, I'd like to start improving this process by inviting folks from
the tempest team to join project's meeting whenever new tests are
going to be worked on.

I'd also like to invite PTLs (or anyone, really) from each team to go
through what's in tempest and what's missing and help this team
improve the test suite. Remember that these tests are also used by the
defcore team and they are not important just for the CI but have an
impact on other areas as well.

I'm doing the above for Glance and I can't stress enough how important
it is for projects to do the same.

Do teams have a different workflow/process to increase tests coverage
in tempest?

Cheers,
Flavio

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Egle Sigler]

Thank you Flavio for bringing this up! We are using tempest tests for
DefCore testing, and we would like to work with anyone willing to increase
coverages in any of the current covered capabilities. We would also like
to hear from the teams when they are planning on removing, changing, or
renaming tests, as that could affect what DefCore tests.

Upcoming DefCore guidelines and tests:
https://github.com/openstack/defcore/blob/master/2016.01.json

Thank you,
Egle

On 12/8/15, 1:25 PM, "Flavio Percoco"  wrote:

>Greetings,
>
>I just reviewed a patch in tempest that proposed adding new tests for
>the Glance's task API. While I believe it's awesome that folks in the
>tempest team keep adding missing tests for projects, I think it'd be
>better for the tempest team, the project's team and defcore if we'd
>discuss these tests before they are worked on. This should help people
>avoid wasting time.
>
>I believe these cases are rare but the benefits of discussing missing
>tests across teams could also help prioritizing the work based on what
>the teams goals are, what the defcore team needs are, etc.
>
>So, I'd like to start improving this process by inviting folks from
>the tempest team to join project's meeting whenever new tests are
>going to be worked on.
>
>I'd also like to invite PTLs (or anyone, really) from each team to go
>through what's in tempest and what's missing and help this team
>improve the test suite. Remember that these tests are also used by the
>defcore team and they are not important just for the CI but have an
>impact on other areas as well.
>
>I'm doing the above for Glance and I can't stress enough how important
>it is for projects to do the same.
>
>Do teams have a different workflow/process to increase tests coverage
>in tempest?
>
>Cheers,
>Flavio
> 
>-- 
>@flaper87
>Flavio Percoco
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Flavio Percoco]

On 08/12/15 22:31 +0100, Jordan Pittier wrote:

Hi Flavio,

On Tue, Dec 8, 2015 at 9:52 PM, Flavio Percoco  wrote:

  
   Oh, I meant ocasionally. Whenever a missing test for an API is found,

   it'd be easy enough for the implementer to sohw up at the meeting and
   bring it up.

From my experience as a Tempest reviewer, I'd say that most newly added tests
are *not* submitted by Tempest regular contributors. I assume (wrongly ?) that
it's mostly people from the actual projects (e.g glance) who are interested in
adding new Tempest tests to test a feature recently implemented. Put
differently, I don't think it's part of Tempest core team/community to add new
tests. We mostly provide a framework and guidance these days.


I agree that the tempest team should focus on providing the framework
rather than the tests themselves. However, these tests are often
contributed by ppl that are not part of the project's team.


But, reading this thread, I don"t know what to suggest. As a Tempest reviewer I
won't start a new ML thread or send a message to a PTL each time I see a new
test being added...I assume the patch author to know what he is doing, I can't
keep on with what's going on in each and every project.


This is what I'd like to avoid. This assumption is exactly what almost
got the tasks API test almost merged and that will likely happen for
other things.

I don't think it's wrong to ping someone from the community when new
tests are added, especially because these tests are used by defcore
as well. Adding the PTL to the review (or some liaison) is simple
enough. We do this for many things in OpenStack. That is, we wait for
PTLs/liaisons approval before going forward with some decisions.


Also, a test can be quickly removed if it is latter on deemed not so useful.


Sure but this is wasting people's time. The contributor's, reviewer's
and community's time as it'll have to be added, reviewed and then
deleted.

I agree this doesn't happen too often but the fact that it happened is
enough of a reason for me to work on improving the process. Again,
especially because these tests are not meant to be used just by our
CI.

Cheers,
Flavio



Jordan
 


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Matthew Treinish]

On Tue, Dec 08, 2015 at 07:25:50PM +, Flavio Percoco wrote:
> Greetings,
> 
> I just reviewed a patch in tempest that proposed adding new tests for
> the Glance's task API. While I believe it's awesome that folks in the
> tempest team keep adding missing tests for projects, I think it'd be
> better for the tempest team, the project's team and defcore if we'd
> discuss these tests before they are worked on. This should help people
> avoid wasting time.

I agree a tasks API test isn't really something we want in tempest. I
actually didn't realize that there was a proposed test for this. (I've
been a bit lax in my tempest reviews lately)

> 
> I believe these cases are rare but the benefits of discussing missing
> tests across teams could also help prioritizing the work based on what
> the teams goals are, what the defcore team needs are, etc.
> 
> So, I'd like to start improving this process by inviting folks from
> the tempest team to join project's meeting whenever new tests are
> going to be worked on.

So, TBH I don't think this will really scale too well. The team of people
dedicated to working on tempest is relatively small compared the size of
OpenStack. I think taking the reverse approach and having project teams
taking a more hands on approach to tempest tests is the only way we'll be
able to scale here.

ML threads like this are actually a good approach on this front I think. It
raises awareness about a proposed test that has potential issues and discusses
it in an archived place.

> 
> I'd also like to invite PTLs (or anyone, really) from each team to go
> through what's in tempest and what's missing and help this team
> improve the test suite. Remember that these tests are also used by the
> defcore team and they are not important just for the CI but have an
> impact on other areas as well.
> 
> I'm doing the above for Glance and I can't stress enough how important
> it is for projects to do the same

A very big +1, on these 2 paragraphs. Tempest is used for much more than gating
and making sure we have what projects view as important is covered is important.
> 
> Do teams have a different workflow/process to increase tests coverage
> in tempest?
> 

I'm not aware of anything going on like this now. At one point in the past as
part of tempest development we used to have a single tempest BP per project per
cycle that project teams would use to collect and track test development for a
cycle. But, I haven't seen that in at least a year, likely longer.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Flavio Percoco]

Greetings,

I just reviewed a patch in tempest that proposed adding new tests for
the Glance's task API. While I believe it's awesome that folks in the
tempest team keep adding missing tests for projects, I think it'd be
better for the tempest team, the project's team and defcore if we'd
discuss these tests before they are worked on. This should help people
avoid wasting time.

I believe these cases are rare but the benefits of discussing missing
tests across teams could also help prioritizing the work based on what
the teams goals are, what the defcore team needs are, etc.

So, I'd like to start improving this process by inviting folks from
the tempest team to join project's meeting whenever new tests are
going to be worked on.

I'd also like to invite PTLs (or anyone, really) from each team to go
through what's in tempest and what's missing and help this team
improve the test suite. Remember that these tests are also used by the
defcore team and they are not important just for the CI but have an
impact on other areas as well.

I'm doing the above for Glance and I can't stress enough how important
it is for projects to do the same.

Do teams have a different workflow/process to increase tests coverage
in tempest?

Cheers,
Flavio

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Flavio Percoco]

On 08/12/15 15:31 -0500, Matthew Treinish wrote:

On Tue, Dec 08, 2015 at 07:25:50PM +, Flavio Percoco wrote:

Greetings,

I just reviewed a patch in tempest that proposed adding new tests for
the Glance's task API. While I believe it's awesome that folks in the
tempest team keep adding missing tests for projects, I think it'd be
better for the tempest team, the project's team and defcore if we'd
discuss these tests before they are worked on. This should help people
avoid wasting time.


I agree a tasks API test isn't really something we want in tempest. I
actually didn't realize that there was a proposed test for this. (I've
been a bit lax in my tempest reviews lately)



I believe these cases are rare but the benefits of discussing missing
tests across teams could also help prioritizing the work based on what
the teams goals are, what the defcore team needs are, etc.

So, I'd like to start improving this process by inviting folks from
the tempest team to join project's meeting whenever new tests are
going to be worked on.


So, TBH I don't think this will really scale too well. The team of people
dedicated to working on tempest is relatively small compared the size of
OpenStack. I think taking the reverse approach and having project teams
taking a more hands on approach to tempest tests is the only way we'll be
able to scale here.


Oh, I meant ocasionally. Whenever a missing test for an API is found,
it'd be easy enough for the implementer to sohw up at the meeting and
bring it up.



ML threads like this are actually a good approach on this front I think. It
raises awareness about a proposed test that has potential issues and discusses
it in an archived place.


This works for me too! :)

Cheers,
Flavio





I'd also like to invite PTLs (or anyone, really) from each team to go
through what's in tempest and what's missing and help this team
improve the test suite. Remember that these tests are also used by the
defcore team and they are not important just for the CI but have an
impact on other areas as well.

I'm doing the above for Glance and I can't stress enough how important
it is for projects to do the same


A very big +1, on these 2 paragraphs. Tempest is used for much more than gating
and making sure we have what projects view as important is covered is important.


Do teams have a different workflow/process to increase tests coverage
in tempest?



I'm not aware of anything going on like this now. At one point in the past as
part of tempest development we used to have a single tempest BP per project per
cycle that project teams would use to collect and track test development for a
cycle. But, I haven't seen that in at least a year, likely longer.

-Matt Treinish




--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest

[Jordan Pittier]

Hi Flavio,

On Tue, Dec 8, 2015 at 9:52 PM, Flavio Percoco  wrote:
>
>
> Oh, I meant ocasionally. Whenever a missing test for an API is found,
> it'd be easy enough for the implementer to sohw up at the meeting and
> bring it up.

>From my experience as a Tempest reviewer, I'd say that most newly added
tests are *not* submitted by Tempest regular contributors. I assume
(wrongly ?) that it's mostly people from the actual projects (e.g glance)
who are interested in adding new Tempest tests to test a feature recently
implemented. Put differently, I don't think it's part of Tempest core
team/community to add new tests. We mostly provide a framework and guidance
these days.

But, reading this thread, I don"t know what to suggest. As a Tempest
reviewer I won't start a new ML thread or send a message to a PTL each time
I see a new test being added...I assume the patch author to know what he is
doing, I can't keep on with what's going on in each and every project.
Also, a test can be quickly removed if it is latter on deemed not so useful.

Jordan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev