Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On Tue, Sep 9, 2014 at 9:58 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 9, 2014, at 10:51 AM, Sean Dague s...@dague.net wrote: On 09/09/2014 10:41 AM, Doug Hellmann wrote: On Sep 8, 2014, at 8:18 PM, James E. Blair cor...@inaugust.com wrote: Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. I don’t think I’ve ever tried to run any of our unit tests on a box where I hadn’t also previously run devstack to install all of those sorts of dependencies. Is that unusual? It is for Linux users, running local unit tests is the norm for me. To be clear, I run the tests on the same host where I ran devstack, not in a VM. I just use devstack as a way to bootstrap all of the libraries needed for the unit test dependencies. I guess I’m just being lazy. :-) You can't run devstack everywhere you code (and want to run tests). I, for example, can't run devstack on my work laptop because I use Gentoo there. And I have MacOS X on my home laptop, so no devstack there too. The latter should be more frequent case in the community. That said I never had a problem with emerging (on either of systems) necessary C libraries for tests to run. As long as they don't pull a lot of (or any) Linux-specific dependencies, it's fine. For me this issue is the case for setuptools' extras. The only problem with them is that we can't specify them in requirement.txt files currently, so we'd have to add another hack to pbr to gather extra dependencies from files like requirements-extra_name.txt or smth like that. Then we can provide different tox venvs for diferent extras sets. -- Kind regards, Yuriy. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On Sep 10, 2014, at 4:34 PM, Yuriy Taraday yorik@gmail.com wrote: On Tue, Sep 9, 2014 at 9:58 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 9, 2014, at 10:51 AM, Sean Dague s...@dague.net wrote: On 09/09/2014 10:41 AM, Doug Hellmann wrote: On Sep 8, 2014, at 8:18 PM, James E. Blair cor...@inaugust.com wrote: Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. I don’t think I’ve ever tried to run any of our unit tests on a box where I hadn’t also previously run devstack to install all of those sorts of dependencies. Is that unusual? It is for Linux users, running local unit tests is the norm for me. To be clear, I run the tests on the same host where I ran devstack, not in a VM. I just use devstack as a way to bootstrap all of the libraries needed for the unit test dependencies. I guess I’m just being lazy. :-) You can't run devstack everywhere you code (and want to run tests). I, for example, can't run devstack on my work laptop because I use Gentoo there. And I have MacOS X on my home laptop, so no devstack there too. The latter should be more frequent case in the community. Sure, I use a Mac but develop OpenStack either on a linux cloud server or a local VM, depending on my internet access. I don’t really see any point in running the tests in an environment we don’t support. Doug That said I never had a problem with emerging (on either of systems) necessary C libraries for tests to run. As long as they don't pull a lot of (or any) Linux-specific dependencies, it's fine. For me this issue is the case for setuptools' extras. The only problem with them is that we can't specify them in requirement.txt files currently, so we'd have to add another hack to pbr to gather extra dependencies from files like requirements-extra_name.txt or smth like that. Then we can provide different tox venvs for diferent extras sets. -- Kind regards, Yuriy. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On Sep 8, 2014, at 8:18 PM, James E. Blair cor...@inaugust.com wrote: Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. I don’t think I’ve ever tried to run any of our unit tests on a box where I hadn’t also previously run devstack to install all of those sorts of dependencies. Is that unusual? Doug We could add another requirements file with heavyweight optional dependencies, and use that in gate testing, but also have a lightweight tox environment that does not include them for ease of use in local testing. What would be really great is if we could use setuptools extras_require for this: https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies However, I'm not sure what the situation is with support for that in pip (and we might need pbr support too). -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On 09/09/2014 10:41 AM, Doug Hellmann wrote: On Sep 8, 2014, at 8:18 PM, James E. Blair cor...@inaugust.com wrote: Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. I don’t think I’ve ever tried to run any of our unit tests on a box where I hadn’t also previously run devstack to install all of those sorts of dependencies. Is that unusual? It is for Linux users, running local unit tests is the norm for me. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On Sep 9, 2014, at 10:51 AM, Sean Dague s...@dague.net wrote: On 09/09/2014 10:41 AM, Doug Hellmann wrote: On Sep 8, 2014, at 8:18 PM, James E. Blair cor...@inaugust.com wrote: Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. I don’t think I’ve ever tried to run any of our unit tests on a box where I hadn’t also previously run devstack to install all of those sorts of dependencies. Is that unusual? It is for Linux users, running local unit tests is the norm for me. To be clear, I run the tests on the same host where I ran devstack, not in a VM. I just use devstack as a way to bootstrap all of the libraries needed for the unit test dependencies. I guess I’m just being lazy. :-) Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
Sean Dague s...@dague.net writes: The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. ... Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. From a technical standpoint, all we need to do to make this work is to add the zookeeper python client bindings to (test-)requirements.txt. But as you point out, that makes it more difficult for developers who want to run unit tests locally without having the requisite libraries and header files installed. We could add another requirements file with heavyweight optional dependencies, and use that in gate testing, but also have a lightweight tox environment that does not include them for ease of use in local testing. What would be really great is if we could use setuptools extras_require for this: https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies However, I'm not sure what the situation is with support for that in pip (and we might need pbr support too). -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On 09/05/2014 07:21 AM, Monty Taylor wrote: On 09/05/2014 06:32 AM, Sean Dague wrote: While reviewing this zookeeper service group fix in Nova - https://review.openstack.org/#/c/102639/ it was exposed that the zookeeper tests aren't running in infra. The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. But it seems like a bad stand off between testing upstream and testing normal path locally. Big picture it would be nice to not require a ton of dev libraries locally for optional components, but still test them upstream. So that in the base case I'm not running zookeeper locally, but if it fails upstream because I broke something in zookeeper, it's easy enough to spin up that dev env that has it. Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. Funny story - I've come to dislike what we're doing here, so I've been slowly working on an idea in this area: https://github.com/emonty/dox The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. It's not quite ready yet - although I'd be happy to move it in to stackforge or even openstack-dev and get other people hacking on it with me until it is. The main problem that needs solving, I think, is how to sanely express multiple target environments (like py26,py27) without making a stupidly baroque config file. OTOH, tox's insistence of making a new virtualenv for each environment is heavyweight and has led to some pretty crazy hacks across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 And then have things like nova build on those, like: infra/nova/py27 Which would have zookeeper as well The _really_ fun part, again, if we can figure out how to express it in config without reimplementing make accidentally, is that we could start to have things like: infra/mysql infra/postgres infra/mongodb And have dox files say things like: Nova unittests want a python27 environment, this means we want an infra/mysql container, an infra/postgres container and for those to be linked to the infra/nova/py27 container where the tests will run. Since those are all reusable, the speed should be _Excellent_ and we should be able to more easily get more things runnable locally without full devstack. Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. I'd like to follow up on this real quick - just to set some expectations appropriately... Firstly, I'm super thrilled that folks are interested in hacking on this - and excited that people have piled on. It's extremely exciting. I want to make sure people know that goal number 1 of this is to allow for a tox-like experience for developers on their laptops that isn't specific to python. We have growing numbers of folks in our numbers who work on things that are not python, and there are a ton of non-OpenStack/non-Python folks out there working on projects that could potentially benefit from something that does for them what tox has so far done for us. Additionally, I personally am bothered by the fact that as a project we declare quite strongly what version of libraries we use that are _python_ but have no current way to do the same for non-python libraries. I may draw the ire ultimately from the distros from where I want my stance on this to get - but I would like very much for us to be able to stop caring about what version of libvirt is _Currently_ in Fedora, RHEL or Ubuntu and get to a point where we _tell_ people that to use OpenStack Kilo they'll need libvirt v7 or something. I think that's a ways off, but I personally want to get there nonetheless. That said - although I'm hopeful it will be, I'd like to be clear that I'm not convinced this will be a useful tool for any of our automation. There are no problems we have in the gate infrastructure that dox is aiming at helping us solve. We already have single-use slaves that already can install anything we need at any time. There are at least four outcomes I can see for the current dox effort: a) dox winds up being a simple and powerful tool for developers both OpenStack and not to use in their day to day life b) a is _so_ true that we feel it's an essential part of openstack developer
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
There was a review once that tried to move the evzookeeper to using kazoo, perhaps that can get reprioritzed?? https://review.openstack.org/#/c/28951/ Sent from my really tiny device... On Sep 5, 2014, at 6:36 AM, Sean Dague s...@dague.net wrote: While reviewing this zookeeper service group fix in Nova - https://review.openstack.org/#/c/102639/ it was exposed that the zookeeper tests aren't running in infra. The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. But it seems like a bad stand off between testing upstream and testing normal path locally. Big picture it would be nice to not require a ton of dev libraries locally for optional components, but still test them upstream. So that in the base case I'm not running zookeeper locally, but if it fails upstream because I broke something in zookeeper, it's easy enough to spin up that dev env that has it. Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On 09/05/2014 06:32 AM, Sean Dague wrote: While reviewing this zookeeper service group fix in Nova - https://review.openstack.org/#/c/102639/ it was exposed that the zookeeper tests aren't running in infra. The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. But it seems like a bad stand off between testing upstream and testing normal path locally. Big picture it would be nice to not require a ton of dev libraries locally for optional components, but still test them upstream. So that in the base case I'm not running zookeeper locally, but if it fails upstream because I broke something in zookeeper, it's easy enough to spin up that dev env that has it. Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. Funny story - I've come to dislike what we're doing here, so I've been slowly working on an idea in this area: https://github.com/emonty/dox The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. It's not quite ready yet - although I'd be happy to move it in to stackforge or even openstack-dev and get other people hacking on it with me until it is. The main problem that needs solving, I think, is how to sanely express multiple target environments (like py26,py27) without making a stupidly baroque config file. OTOH, tox's insistence of making a new virtualenv for each environment is heavyweight and has led to some pretty crazy hacks across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 And then have things like nova build on those, like: infra/nova/py27 Which would have zookeeper as well The _really_ fun part, again, if we can figure out how to express it in config without reimplementing make accidentally, is that we could start to have things like: infra/mysql infra/postgres infra/mongodb And have dox files say things like: Nova unittests want a python27 environment, this means we want an infra/mysql container, an infra/postgres container and for those to be linked to the infra/nova/py27 container where the tests will run. Since those are all reusable, the speed should be _Excellent_ and we should be able to more easily get more things runnable locally without full devstack. Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On 09/05/2014 04:21 PM, Monty Taylor wrote: On 09/05/2014 06:32 AM, Sean Dague wrote: While reviewing this zookeeper service group fix in Nova - https://review.openstack.org/#/c/102639/ it was exposed that the zookeeper tests aren't running in infra. The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. But it seems like a bad stand off between testing upstream and testing normal path locally. Big picture it would be nice to not require a ton of dev libraries locally for optional components, but still test them upstream. So that in the base case I'm not running zookeeper locally, but if it fails upstream because I broke something in zookeeper, it's easy enough to spin up that dev env that has it. Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. Funny story - I've come to dislike what we're doing here, so I've been slowly working on an idea in this area: https://github.com/emonty/dox The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. It's not quite ready yet - although I'd be happy to move it in to stackforge or even openstack-dev and get other people hacking on it with me until it is. The main problem that needs solving, I think, is how to sanely express multiple target environments (like py26,py27) without making a stupidly baroque config file. OTOH, tox's insistence of making a new virtualenv for each environment is heavyweight and has led to some pretty crazy hacks across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 And then have things like nova build on those, like: infra/nova/py27 Which would have zookeeper as well The _really_ fun part, again, if we can figure out how to express it in config without reimplementing make accidentally, is that we could start to have things like: infra/mysql infra/postgres infra/mongodb And have dox files say things like: Nova unittests want a python27 environment, this means we want an infra/mysql container, an infra/postgres container and for those to be linked to the infra/nova/py27 container where the tests will run. Since those are all reusable, the speed should be _Excellent_ and we should be able to more easily get more things runnable locally without full devstack. Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. I think it is sexy - I don't describe ideas/software as sexy that often but this one deserves it. I'm interested in helping out. I'll clone it and give it a try - or at least take a look at it. Flavio Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
I agree with Flavio. This looks really cool, and I had a very similar idea recently. I'll try to find some time to give this a whirl. -Dave On Fri, Sep 5, 2014 at 10:37 AM, Flavio Percoco fla...@redhat.com wrote: On 09/05/2014 04:21 PM, Monty Taylor wrote: On 09/05/2014 06:32 AM, Sean Dague wrote: While reviewing this zookeeper service group fix in Nova - https://review.openstack.org/#/c/102639/ it was exposed that the zookeeper tests aren't running in infra. The crux of the issue is that zookeeper python modules are C extensions. So you have to either install from packages (which we don't do in unit tests) or install from pip, which means forcing zookeeper dev packages locally. Realistically this is the same issue we end up with for mysql and pg, but given their wider usage we just forced that pain on developers. But it seems like a bad stand off between testing upstream and testing normal path locally. Big picture it would be nice to not require a ton of dev libraries locally for optional components, but still test them upstream. So that in the base case I'm not running zookeeper locally, but if it fails upstream because I broke something in zookeeper, it's easy enough to spin up that dev env that has it. Which feels like we need some decoupling on our requirements vs. tox targets to get there. CC to Monty and Clark as our super awesome tox hackers to help figure out if there is a path forward here that makes sense. Funny story - I've come to dislike what we're doing here, so I've been slowly working on an idea in this area: https://github.com/emonty/dox The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. It's not quite ready yet - although I'd be happy to move it in to stackforge or even openstack-dev and get other people hacking on it with me until it is. The main problem that needs solving, I think, is how to sanely express multiple target environments (like py26,py27) without making a stupidly baroque config file. OTOH, tox's insistence of making a new virtualenv for each environment is heavyweight and has led to some pretty crazy hacks across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 And then have things like nova build on those, like: infra/nova/py27 Which would have zookeeper as well The _really_ fun part, again, if we can figure out how to express it in config without reimplementing make accidentally, is that we could start to have things like: infra/mysql infra/postgres infra/mongodb And have dox files say things like: Nova unittests want a python27 environment, this means we want an infra/mysql container, an infra/postgres container and for those to be linked to the infra/nova/py27 container where the tests will run. Since those are all reusable, the speed should be _Excellent_ and we should be able to more easily get more things runnable locally without full devstack. Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. I think it is sexy - I don't describe ideas/software as sexy that often but this one deserves it. I'm interested in helping out. I'll clone it and give it a try - or at least take a look at it. Flavio Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- David Shrewsbury (Shrews) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On Fri, 5 Sep 2014, Monty Taylor wrote: The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. Oh thank god[1]. Seriously. jogo started a thread about what matters for kilo and I was going to respond with (amongst other things) get containers into the testing scene. Seems you're way ahead of me. Docker's caching could be a _huge_ win here. [1] https://www.youtube.com/watch?v=om5rbtudzrg across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 I'm assuming these would get rebuilt regularly (every time global requirements and friends are updated) on some sort of automated hook? Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. Given availability (currently an unknown) I'd like to help with this. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
On 09/05/2014 12:05 PM, Chris Dent wrote: On Fri, 5 Sep 2014, Monty Taylor wrote: The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. Oh thank god[1]. Seriously. jogo started a thread about what matters for kilo and I was going to respond with (amongst other things) get containers into the testing scene. Seems you're way ahead of me. Docker's caching could be a _huge_ win here. [1] https://www.youtube.com/watch?v=om5rbtudzrg across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 I'm assuming these would get rebuilt regularly (every time global requirements and friends are updated) on some sort of automated hook? Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. Given availability (currently an unknown) I'd like to help with this. I think all this is very cool, I'd say if we're going to put it in gerrit do stackforge instead of openstack/ namespace, because it will be useful beyond us. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to provide tests environments for python things that require C extensions
I too agree this can be useful beyond openstack. It will great if one of you (experts) can explain in more details how the python virtual environment is heavyweight than docker containers. I am just a user of devstack without the nitty gritty details of its inner workings. However, I can say that sometimes my Ubuntu VM that has devstack tempest tests running exhaust the entire CPU (make the machine unusable) after the tests are run. Regards, Amit *CloudByte Inc.* http://www.cloudbyte.com/ On Fri, Sep 5, 2014 at 9:44 PM, Sean Dague s...@dague.net wrote: On 09/05/2014 12:05 PM, Chris Dent wrote: On Fri, 5 Sep 2014, Monty Taylor wrote: The tl;dr is it's like tox, except it uses docker instead of virtualenv - which means we can express all of our requirements, not just pip ones. Oh thank god[1]. Seriously. jogo started a thread about what matters for kilo and I was going to respond with (amongst other things) get containers into the testing scene. Seems you're way ahead of me. Docker's caching could be a _huge_ win here. [1] https://www.youtube.com/watch?v=om5rbtudzrg across the project. Luckily, docker itself does an EXCELLENT job at handling caching and reuse - so I think we can have a set of containers that something in infra (waves hands) publishes to dockerhub, like: infra/py27 infra/py26 I'm assuming these would get rebuilt regularly (every time global requirements and friends are updated) on some sort of automated hook? Thoughts? Anybody wanna hack on it with me? I think it could wind up being a pretty useful tool for folks outside of OpenStack too if we get it right. Given availability (currently an unknown) I'd like to help with this. I think all this is very cool, I'd say if we're going to put it in gerrit do stackforge instead of openstack/ namespace, because it will be useful beyond us. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
