Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
On 17/12/15 02:00 +, Egle Sigler wrote: Thank you Flavio for bringing this up! We are using tempest tests for DefCore testing, and we would like to work with anyone willing to increase coverages in any of the current covered capabilities. We would also like to hear from the teams when they are planning on removing, changing, or renaming tests, as that could affect what DefCore tests. Upcoming DefCore guidelines and tests: https://github.com/openstack/defcore/blob/master/2016.01.json +1 It's taking me a bit longer than I expeced but I'm working on a list of tests that would be great to have in tempest and some that would perhaps be better to remove. Thanks for the feedback, Flavio Thank you, Egle On 12/8/15, 1:25 PM, "Flavio Percoco"wrote: Greetings, I just reviewed a patch in tempest that proposed adding new tests for the Glance's task API. While I believe it's awesome that folks in the tempest team keep adding missing tests for projects, I think it'd be better for the tempest team, the project's team and defcore if we'd discuss these tests before they are worked on. This should help people avoid wasting time. I believe these cases are rare but the benefits of discussing missing tests across teams could also help prioritizing the work based on what the teams goals are, what the defcore team needs are, etc. So, I'd like to start improving this process by inviting folks from the tempest team to join project's meeting whenever new tests are going to be worked on. I'd also like to invite PTLs (or anyone, really) from each team to go through what's in tempest and what's missing and help this team improve the test suite. Remember that these tests are also used by the defcore team and they are not important just for the CI but have an impact on other areas as well. I'm doing the above for Glance and I can't stress enough how important it is for projects to do the same. Do teams have a different workflow/process to increase tests coverage in tempest? Cheers, Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
Thank you Flavio for bringing this up! We are using tempest tests for DefCore testing, and we would like to work with anyone willing to increase coverages in any of the current covered capabilities. We would also like to hear from the teams when they are planning on removing, changing, or renaming tests, as that could affect what DefCore tests. Upcoming DefCore guidelines and tests: https://github.com/openstack/defcore/blob/master/2016.01.json Thank you, Egle On 12/8/15, 1:25 PM, "Flavio Percoco"wrote: >Greetings, > >I just reviewed a patch in tempest that proposed adding new tests for >the Glance's task API. While I believe it's awesome that folks in the >tempest team keep adding missing tests for projects, I think it'd be >better for the tempest team, the project's team and defcore if we'd >discuss these tests before they are worked on. This should help people >avoid wasting time. > >I believe these cases are rare but the benefits of discussing missing >tests across teams could also help prioritizing the work based on what >the teams goals are, what the defcore team needs are, etc. > >So, I'd like to start improving this process by inviting folks from >the tempest team to join project's meeting whenever new tests are >going to be worked on. > >I'd also like to invite PTLs (or anyone, really) from each team to go >through what's in tempest and what's missing and help this team >improve the test suite. Remember that these tests are also used by the >defcore team and they are not important just for the CI but have an >impact on other areas as well. > >I'm doing the above for Glance and I can't stress enough how important >it is for projects to do the same. > >Do teams have a different workflow/process to increase tests coverage >in tempest? > >Cheers, >Flavio > >-- >@flaper87 >Flavio Percoco > >__ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
On 08/12/15 22:31 +0100, Jordan Pittier wrote: Hi Flavio, On Tue, Dec 8, 2015 at 9:52 PM, Flavio Percocowrote: Oh, I meant ocasionally. Whenever a missing test for an API is found, it'd be easy enough for the implementer to sohw up at the meeting and bring it up. From my experience as a Tempest reviewer, I'd say that most newly added tests are *not* submitted by Tempest regular contributors. I assume (wrongly ?) that it's mostly people from the actual projects (e.g glance) who are interested in adding new Tempest tests to test a feature recently implemented. Put differently, I don't think it's part of Tempest core team/community to add new tests. We mostly provide a framework and guidance these days. I agree that the tempest team should focus on providing the framework rather than the tests themselves. However, these tests are often contributed by ppl that are not part of the project's team. But, reading this thread, I don"t know what to suggest. As a Tempest reviewer I won't start a new ML thread or send a message to a PTL each time I see a new test being added...I assume the patch author to know what he is doing, I can't keep on with what's going on in each and every project. This is what I'd like to avoid. This assumption is exactly what almost got the tasks API test almost merged and that will likely happen for other things. I don't think it's wrong to ping someone from the community when new tests are added, especially because these tests are used by defcore as well. Adding the PTL to the review (or some liaison) is simple enough. We do this for many things in OpenStack. That is, we wait for PTLs/liaisons approval before going forward with some decisions. Also, a test can be quickly removed if it is latter on deemed not so useful. Sure but this is wasting people's time. The contributor's, reviewer's and community's time as it'll have to be added, reviewed and then deleted. I agree this doesn't happen too often but the fact that it happened is enough of a reason for me to work on improving the process. Again, especially because these tests are not meant to be used just by our CI. Cheers, Flavio Jordan -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
On Tue, Dec 08, 2015 at 07:25:50PM +, Flavio Percoco wrote: > Greetings, > > I just reviewed a patch in tempest that proposed adding new tests for > the Glance's task API. While I believe it's awesome that folks in the > tempest team keep adding missing tests for projects, I think it'd be > better for the tempest team, the project's team and defcore if we'd > discuss these tests before they are worked on. This should help people > avoid wasting time. I agree a tasks API test isn't really something we want in tempest. I actually didn't realize that there was a proposed test for this. (I've been a bit lax in my tempest reviews lately) > > I believe these cases are rare but the benefits of discussing missing > tests across teams could also help prioritizing the work based on what > the teams goals are, what the defcore team needs are, etc. > > So, I'd like to start improving this process by inviting folks from > the tempest team to join project's meeting whenever new tests are > going to be worked on. So, TBH I don't think this will really scale too well. The team of people dedicated to working on tempest is relatively small compared the size of OpenStack. I think taking the reverse approach and having project teams taking a more hands on approach to tempest tests is the only way we'll be able to scale here. ML threads like this are actually a good approach on this front I think. It raises awareness about a proposed test that has potential issues and discusses it in an archived place. > > I'd also like to invite PTLs (or anyone, really) from each team to go > through what's in tempest and what's missing and help this team > improve the test suite. Remember that these tests are also used by the > defcore team and they are not important just for the CI but have an > impact on other areas as well. > > I'm doing the above for Glance and I can't stress enough how important > it is for projects to do the same A very big +1, on these 2 paragraphs. Tempest is used for much more than gating and making sure we have what projects view as important is covered is important. > > Do teams have a different workflow/process to increase tests coverage > in tempest? > I'm not aware of anything going on like this now. At one point in the past as part of tempest development we used to have a single tempest BP per project per cycle that project teams would use to collect and track test development for a cycle. But, I haven't seen that in at least a year, likely longer. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
Greetings, I just reviewed a patch in tempest that proposed adding new tests for the Glance's task API. While I believe it's awesome that folks in the tempest team keep adding missing tests for projects, I think it'd be better for the tempest team, the project's team and defcore if we'd discuss these tests before they are worked on. This should help people avoid wasting time. I believe these cases are rare but the benefits of discussing missing tests across teams could also help prioritizing the work based on what the teams goals are, what the defcore team needs are, etc. So, I'd like to start improving this process by inviting folks from the tempest team to join project's meeting whenever new tests are going to be worked on. I'd also like to invite PTLs (or anyone, really) from each team to go through what's in tempest and what's missing and help this team improve the test suite. Remember that these tests are also used by the defcore team and they are not important just for the CI but have an impact on other areas as well. I'm doing the above for Glance and I can't stress enough how important it is for projects to do the same. Do teams have a different workflow/process to increase tests coverage in tempest? Cheers, Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
On 08/12/15 15:31 -0500, Matthew Treinish wrote: On Tue, Dec 08, 2015 at 07:25:50PM +, Flavio Percoco wrote: Greetings, I just reviewed a patch in tempest that proposed adding new tests for the Glance's task API. While I believe it's awesome that folks in the tempest team keep adding missing tests for projects, I think it'd be better for the tempest team, the project's team and defcore if we'd discuss these tests before they are worked on. This should help people avoid wasting time. I agree a tasks API test isn't really something we want in tempest. I actually didn't realize that there was a proposed test for this. (I've been a bit lax in my tempest reviews lately) I believe these cases are rare but the benefits of discussing missing tests across teams could also help prioritizing the work based on what the teams goals are, what the defcore team needs are, etc. So, I'd like to start improving this process by inviting folks from the tempest team to join project's meeting whenever new tests are going to be worked on. So, TBH I don't think this will really scale too well. The team of people dedicated to working on tempest is relatively small compared the size of OpenStack. I think taking the reverse approach and having project teams taking a more hands on approach to tempest tests is the only way we'll be able to scale here. Oh, I meant ocasionally. Whenever a missing test for an API is found, it'd be easy enough for the implementer to sohw up at the meeting and bring it up. ML threads like this are actually a good approach on this front I think. It raises awareness about a proposed test that has potential issues and discusses it in an archived place. This works for me too! :) Cheers, Flavio I'd also like to invite PTLs (or anyone, really) from each team to go through what's in tempest and what's missing and help this team improve the test suite. Remember that these tests are also used by the defcore team and they are not important just for the CI but have an impact on other areas as well. I'm doing the above for Glance and I can't stress enough how important it is for projects to do the same A very big +1, on these 2 paragraphs. Tempest is used for much more than gating and making sure we have what projects view as important is covered is important. Do teams have a different workflow/process to increase tests coverage in tempest? I'm not aware of anything going on like this now. At one point in the past as part of tempest development we used to have a single tempest BP per project per cycle that project teams would use to collect and track test development for a cycle. But, I haven't seen that in at least a year, likely longer. -Matt Treinish -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][tempest][defcore] Process to imrpove tests coverge in temepest
Hi Flavio, On Tue, Dec 8, 2015 at 9:52 PM, Flavio Percocowrote: > > > Oh, I meant ocasionally. Whenever a missing test for an API is found, > it'd be easy enough for the implementer to sohw up at the meeting and > bring it up. >From my experience as a Tempest reviewer, I'd say that most newly added tests are *not* submitted by Tempest regular contributors. I assume (wrongly ?) that it's mostly people from the actual projects (e.g glance) who are interested in adding new Tempest tests to test a feature recently implemented. Put differently, I don't think it's part of Tempest core team/community to add new tests. We mostly provide a framework and guidance these days. But, reading this thread, I don"t know what to suggest. As a Tempest reviewer I won't start a new ML thread or send a message to a PTL each time I see a new test being added...I assume the patch author to know what he is doing, I can't keep on with what's going on in each and every project. Also, a test can be quickly removed if it is latter on deemed not so useful. Jordan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev