Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On Wed, 2014-02-12 at 14:17 -0800, Maru Newby wrote: On Feb 12, 2014, at 1:59 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 04:25 PM, Maru Newby wrote: On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Our direct experience is that if we don't do this, within 2 weeks some project will have landed API breaking changes. This approach actually takes a lot of review load off the core reviewers, so reverting to a model which puts more work back on the review team (given the current review load), isn't something I think we want. Just so I'm clear, is there anything I could say that would change your mind? I'd like to discuss this at tomorrow's IRC meeting, if possible. I think that the model that Maru came up with (PoC code in the two patches referenced above) are actually a pretty slick way of dealing with this issue, and along with the ongoing efforts to libify Tempest, I think that we should head in this direction if possible. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][neutron] API tests in the Neutron tree
At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Today most projects (excepting swift... which I'll get to in a second) think about testing in 2 ways. Unit tests driven by tox in a venv, and tempest tests in a devstack environment. Because of this dualism, people have put increasingly more awkward live environments in the tox unit tests jobs. For instance, IIRC, neutron actually starts a full wsgi stack to tests every single in tree plugin, instead of just testing the driver call down path. Swift did something a little different. They have 3 classes of things. Unit tests, Tempest Tests, and Swift Functional tests. The Swift functional tests run in a devstack, but not with Tempest. Instead they run their own suite. This test suite only runs on the swift project, not on other projects, and it's something they can use to test functional scenarios that wouldn't fit in tempest, and extend to their heart's content, not having to worry about the interaction with other components, because it's only pushing on swift. Going down this third path with neutron testing is I think very valuable. Honestly, I'd like to encourage more projects to do this as well. It would give them a greater component assuredness before entering the integrated gate. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Today most projects (excepting swift... which I'll get to in a second) think about testing in 2 ways. Unit tests driven by tox in a venv, and tempest tests in a devstack environment. Because of this dualism, people have put increasingly more awkward live environments in the tox unit tests jobs. For instance, IIRC, neutron actually starts a full wsgi stack to tests every single in tree plugin, instead of just testing the driver call down path. Yeah, and this full-stack approach means that neutron tests are incredibly hard to maintain and debug, to say nothing of the time penalty (approaching the duration of a tempest smoke run). Part of the benefit of the proposal in question is to allow targeting the plugins directly with the same tests that will run against the full stack. m. Swift did something a little different. They have 3 classes of things. Unit tests, Tempest Tests, and Swift Functional tests. The Swift functional tests run in a devstack, but not with Tempest. Instead they run their own suite. This test suite only runs on the swift project, not on other projects, and it's something they can use to test functional scenarios that wouldn't fit in tempest, and extend to their heart's content, not having to worry about the interaction with other components, because it's only pushing on swift. Going down this third path with neutron testing is I think very valuable. Honestly, I'd like to encourage more projects to do this as well. It would give them a greater component assuredness before entering the integrated gate. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On 02/12/2014 04:25 PM, Maru Newby wrote: On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Our direct experience is that if we don't do this, within 2 weeks some project will have landed API breaking changes. This approach actually takes a lot of review load off the core reviewers, so reverting to a model which puts more work back on the review team (given the current review load), isn't something I think we want. I get that there is a cost with this. But there is a cost of all of this. And because API tests should be write once for the API (and specifically not evolving), the upfront cost vs. the continually paid cost of review time in tree has been the better trade off. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On Feb 12, 2014, at 1:59 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 04:25 PM, Maru Newby wrote: On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Our direct experience is that if we don't do this, within 2 weeks some project will have landed API breaking changes. This approach actually takes a lot of review load off the core reviewers, so reverting to a model which puts more work back on the review team (given the current review load), isn't something I think we want. Just so I'm clear, is there anything I could say that would change your mind? m. I get that there is a cost with this. But there is a cost of all of this. And because API tests should be write once for the API (and specifically not evolving), the upfront cost vs. the continually paid cost of review time in tree has been the better trade off. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
