Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
Hi, On Thu, Mar 26, 2015 at 10:26 PM, Zane Bitter zbit...@redhat.com wrote: On 26/03/15 10:38, Pavlo Shchelokovskyy wrote: Hi all, following IRC discussion here is a summary of what I propose can be done in this regard, in the order of increased decoupling: 1) make a separate requirements.txt for integration tests and modify the tox job to use it. The code of these tests is pretty much decoupled already, not using any modules from the main heat tree. The actual dependencies are mostly api clients and test framework. Making this happen should decrease the time needed to setup the tox env and thus speed up the test run somewhat. +1 2) provide separate distutils' setup.py/setup.cfg http://setup.py/setup.cfg to ease packaging and installing this test suit to run it against an already deployed cloud (especially scenario tests seem to be valuable in this regard). +1 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. -0 I'm not sure what the advantage is here, and there are a bunch of downsides (basically, I agree with Ryan). Unfortunately I missed the IRC discussion, can you elaborate on how decoupling to this degree might help us? Presumably this could enable a more streamlined packaging and publishing of the test suit (e.g. to PyPI). But I agree, right now it is not really needed given the downsides, I just brought it up as an extreme separation case to collect more opinions. Given the feedback we have in the thread, I will proceed with the first point as this should have immediate benefit for the duration of the test job and already give help to those who want to package the test suit separately. Distutils stuff can be added later. Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
Hello, As a QA engineer, I like the idea to make integration tests more independent and have an ability to package them and run against any deployed cloud, it will be very useful. But I assume, that creating a separate repository is not needed and it is enough to just collect all functional/integration tests in separate folder like now. Best regards, Anastasia Kuznetsova On Fri, Mar 27, 2015 at 6:18 AM, Angus Salkeld asalk...@mirantis.com wrote: On Fri, Mar 27, 2015 at 6:26 AM, Zane Bitter zbit...@redhat.com wrote: On 26/03/15 10:38, Pavlo Shchelokovskyy wrote: Hi all, following IRC discussion here is a summary of what I propose can be done in this regard, in the order of increased decoupling: 1) make a separate requirements.txt for integration tests and modify the tox job to use it. The code of these tests is pretty much decoupled already, not using any modules from the main heat tree. The actual dependencies are mostly api clients and test framework. Making this happen should decrease the time needed to setup the tox env and thus speed up the test run somewhat. +1 2) provide separate distutils' setup.py/setup.cfg http://setup.py/setup.cfg to ease packaging and installing this test suit to run it against an already deployed cloud (especially scenario tests seem to be valuable in this regard). +1 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. -0 I'm not sure what the advantage is here, and there are a bunch of downsides (basically, I agree with Ryan). Unfortunately I missed the IRC discussion, can you elaborate on how decoupling to this degree might help us? I think the overall goal is to make it easier for an operator to run tests against their cloud to make sure everything is working. We should really have a common approach to this so they don't have to do something different for each project. Any opinions from the QA team? Maybe have it as it's own package, then you can install it and run something like: os-functional-tests-run package-name auth args here -A cheers, Zane. What do you think about it? Please share your comments. Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com http://www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
On 03/27/2015 06:57 AM, Pavlo Shchelokovskyy wrote: Hi, On Thu, Mar 26, 2015 at 10:26 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: [snip] 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. -0 I'm not sure what the advantage is here, and there are a bunch of downsides (basically, I agree with Ryan). Unfortunately I missed the IRC discussion, can you elaborate on how decoupling to this degree might help us? Presumably this could enable a more streamlined packaging and publishing of the test suit (e.g. to PyPI). But I agree, right now it is not really needed given the downsides, I just brought it up as an extreme separation case to collect more opinions. Given the feedback we have in the thread, I will proceed with the first point as this should have immediate benefit for the duration of the test job and already give help to those who want to package the test suit separately. Distutils stuff can be added later. Best regards, Pavlo Shchelokovskyy If we only do 1 and 2, not 3 we get all the benefits (separate package, streamlined publishing, etc) without having to deal with the submodule disadvantages I (and you) mentioned earlier. -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
On Fri, Mar 27, 2015 at 6:26 AM, Zane Bitter zbit...@redhat.com wrote: On 26/03/15 10:38, Pavlo Shchelokovskyy wrote: Hi all, following IRC discussion here is a summary of what I propose can be done in this regard, in the order of increased decoupling: 1) make a separate requirements.txt for integration tests and modify the tox job to use it. The code of these tests is pretty much decoupled already, not using any modules from the main heat tree. The actual dependencies are mostly api clients and test framework. Making this happen should decrease the time needed to setup the tox env and thus speed up the test run somewhat. +1 2) provide separate distutils' setup.py/setup.cfg http://setup.py/setup.cfg to ease packaging and installing this test suit to run it against an already deployed cloud (especially scenario tests seem to be valuable in this regard). +1 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. -0 I'm not sure what the advantage is here, and there are a bunch of downsides (basically, I agree with Ryan). Unfortunately I missed the IRC discussion, can you elaborate on how decoupling to this degree might help us? I think the overall goal is to make it easier for an operator to run tests against their cloud to make sure everything is working. We should really have a common approach to this so they don't have to do something different for each project. Any opinions from the QA team? Maybe have it as it's own package, then you can install it and run something like: os-functional-tests-run package-name auth args here -A cheers, Zane. What do you think about it? Please share your comments. Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com http://www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
On 03/26/2015 10:38 AM, Pavlo Shchelokovskyy wrote: Hi all, following IRC discussion here is a summary of what I propose can be done in this regard, in the order of increased decoupling: 1) make a separate requirements.txt for integration tests and modify the tox job to use it. The code of these tests is pretty much decoupled already, not using any modules from the main heat tree. The actual dependencies are mostly api clients and test framework. Making this happen should decrease the time needed to setup the tox env and thus speed up the test run somewhat. +1 for this 2) provide separate distutils' setup.py/setup.cfg to ease packaging and installing this test suit to run it against an already deployed cloud (especially scenario tests seem to be valuable in this regard). I quite like this idea, the value here is pretty apparent in the spirit of the separate requirements.txt. 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. It's also important for local development workflow to have an up-to-date version of the project's tests and having them shuffled out to a submodule would make it exceptionally easy to forget submodule pull and end up missing tests. This is, of course, in addition to your (all valid) reasons for avoiding submodules. What do you think about it? Please share your comments. Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com http://www.mirantis.com -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Decoupling Heat integration tests from Heat tree
On 26/03/15 10:38, Pavlo Shchelokovskyy wrote: Hi all, following IRC discussion here is a summary of what I propose can be done in this regard, in the order of increased decoupling: 1) make a separate requirements.txt for integration tests and modify the tox job to use it. The code of these tests is pretty much decoupled already, not using any modules from the main heat tree. The actual dependencies are mostly api clients and test framework. Making this happen should decrease the time needed to setup the tox env and thus speed up the test run somewhat. +1 2) provide separate distutils' setup.py/setup.cfg http://setup.py/setup.cfg to ease packaging and installing this test suit to run it against an already deployed cloud (especially scenario tests seem to be valuable in this regard). +1 3) move the integration tests to a separate repo and use it as git submodule in the main tree. The main reasons not to do it as far as I've collected are not being able to provide code change and test in the same (or dependent) commits, and lesser reviewers' attention to a separate repo. -0 I'm not sure what the advantage is here, and there are a bunch of downsides (basically, I agree with Ryan). Unfortunately I missed the IRC discussion, can you elaborate on how decoupling to this degree might help us? cheers, Zane. What do you think about it? Please share your comments. Best regards, Pavlo Shchelokovskyy Software Engineer Mirantis Inc www.mirantis.com http://www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
