Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests
-Original Message- From: Matthew Treinish [mailto:mtrein...@kortar.org] Sent: Wednesday, June 18, 2014 10:33 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests On Tue, Jun 17, 2014 at 01:45:55AM +, Kenichi Oomichi wrote: -Original Message- From: Matthew Treinish [mailto:mtrein...@kortar.org] Sent: Monday, June 16, 2014 11:58 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests On Mon, Jun 16, 2014 at 10:46:51AM -0400, David Kranz wrote: I have been reviewing some of these specs and sense a lack of clarity around what is expected. In the pre-qa-specs world we did not want tempest blueprints to be used by projects to track their tempest test submissions because the core review team did not want to have to spend a lot of time dealing with that. We said that each project could have one tempest blueprint that would point to some other place (project blueprints, spreadsheet, etherpad, etc.) that would track specific tests to be added. I'm not sure what aspect of the new qa-spec process would make us feel differently about this. Has this policy changed? We should spell out the expectation in any event. I will update the README when we have a conclusion. The policy has not changed. There should be 1 BP (or maybe 2 or 3 if they want to split the effort a bit more granularly for tracking different classes of tests, but still 1 BP series) for improving project tests. For individual tests part of a bigger effort should be tracked outside of the Tempest LP. IMO after it's approved the spec/BP for tracking test additions is only really useful to have a unified topic to use for review classification. +1 to use a single blueprint for adding new tests of each project. The unified topic of each project would be useful to get each project reviewers' effort on the Tempest tests reviews. To add new tests, do we need to have qa-specs, or is it OK to have blueprints only? So I've been asking all the new BPs for project testing being opened this cycle to have a spec too. My feeling is that we should only have one process for doing BPs/specs that way we get all the artifacts in the same place. It should also hopefully get everyone more involved with the qa-specs workflow. I see, thank you for clarifying it. The specs for adding project test should be pretty simple, they just basically need to outline what project is going to be tested, what types of tests are going to be worked on, (API, CLI, etc..) and how the test development is going to be tracked. (etherpad, google doc, etc.) That is nice advice, it seems easy to review also :-) Thanks Ken'ichi Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests
On Tue, Jun 17, 2014 at 01:45:55AM +, Kenichi Oomichi wrote: -Original Message- From: Matthew Treinish [mailto:mtrein...@kortar.org] Sent: Monday, June 16, 2014 11:58 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests On Mon, Jun 16, 2014 at 10:46:51AM -0400, David Kranz wrote: I have been reviewing some of these specs and sense a lack of clarity around what is expected. In the pre-qa-specs world we did not want tempest blueprints to be used by projects to track their tempest test submissions because the core review team did not want to have to spend a lot of time dealing with that. We said that each project could have one tempest blueprint that would point to some other place (project blueprints, spreadsheet, etherpad, etc.) that would track specific tests to be added. I'm not sure what aspect of the new qa-spec process would make us feel differently about this. Has this policy changed? We should spell out the expectation in any event. I will update the README when we have a conclusion. The policy has not changed. There should be 1 BP (or maybe 2 or 3 if they want to split the effort a bit more granularly for tracking different classes of tests, but still 1 BP series) for improving project tests. For individual tests part of a bigger effort should be tracked outside of the Tempest LP. IMO after it's approved the spec/BP for tracking test additions is only really useful to have a unified topic to use for review classification. +1 to use a single blueprint for adding new tests of each project. The unified topic of each project would be useful to get each project reviewers' effort on the Tempest tests reviews. To add new tests, do we need to have qa-specs, or is it OK to have blueprints only? So I've been asking all the new BPs for project testing being opened this cycle to have a spec too. My feeling is that we should only have one process for doing BPs/specs that way we get all the artifacts in the same place. It should also hopefully get everyone more involved with the qa-specs workflow. The specs for adding project test should be pretty simple, they just basically need to outline what project is going to be tested, what types of tests are going to be worked on, (API, CLI, etc..) and how the test development is going to be tracked. (etherpad, google doc, etc.) -Matt Treinish pgpve1uKjCslX.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests
I have been reviewing some of these specs and sense a lack of clarity around what is expected. In the pre-qa-specs world we did not want tempest blueprints to be used by projects to track their tempest test submissions because the core review team did not want to have to spend a lot of time dealing with that. We said that each project could have one tempest blueprint that would point to some other place (project blueprints, spreadsheet, etherpad, etc.) that would track specific tests to be added. I'm not sure what aspect of the new qa-spec process would make us feel differently about this. Has this policy changed? We should spell out the expectation in any event. I will update the README when we have a conclusion. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests
On Mon, Jun 16, 2014 at 10:46:51AM -0400, David Kranz wrote: I have been reviewing some of these specs and sense a lack of clarity around what is expected. In the pre-qa-specs world we did not want tempest blueprints to be used by projects to track their tempest test submissions because the core review team did not want to have to spend a lot of time dealing with that. We said that each project could have one tempest blueprint that would point to some other place (project blueprints, spreadsheet, etherpad, etc.) that would track specific tests to be added. I'm not sure what aspect of the new qa-spec process would make us feel differently about this. Has this policy changed? We should spell out the expectation in any event. I will update the README when we have a conclusion. The policy has not changed. There should be 1 BP (or maybe 2 or 3 if they want to split the effort a bit more granularly for tracking different classes of tests, but still 1 BP series) for improving project tests. For individual tests part of a bigger effort should be tracked outside of the Tempest LP. IMO after it's approved the spec/BP for tracking test additions is only really useful to have a unified topic to use for review classification. Also, to be fair I'm not sure things were clear before we moved to specs. When I cleaned up the BP list at the beginning of the cycle there were a couple of BPs on the list which didn't conform to this. -Matt Treinish pgp7KB5jXHLqU.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests
-Original Message- From: Matthew Treinish [mailto:mtrein...@kortar.org] Sent: Monday, June 16, 2014 11:58 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [qa] Clarification of policy for qa-specs around adding new tests On Mon, Jun 16, 2014 at 10:46:51AM -0400, David Kranz wrote: I have been reviewing some of these specs and sense a lack of clarity around what is expected. In the pre-qa-specs world we did not want tempest blueprints to be used by projects to track their tempest test submissions because the core review team did not want to have to spend a lot of time dealing with that. We said that each project could have one tempest blueprint that would point to some other place (project blueprints, spreadsheet, etherpad, etc.) that would track specific tests to be added. I'm not sure what aspect of the new qa-spec process would make us feel differently about this. Has this policy changed? We should spell out the expectation in any event. I will update the README when we have a conclusion. The policy has not changed. There should be 1 BP (or maybe 2 or 3 if they want to split the effort a bit more granularly for tracking different classes of tests, but still 1 BP series) for improving project tests. For individual tests part of a bigger effort should be tracked outside of the Tempest LP. IMO after it's approved the spec/BP for tracking test additions is only really useful to have a unified topic to use for review classification. +1 to use a single blueprint for adding new tests of each project. The unified topic of each project would be useful to get each project reviewers' effort on the Tempest tests reviews. To add new tests, do we need to have qa-specs, or is it OK to have blueprints only? Thanks Ken'ichi Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
