Re: [openstack-dev] [TripleO] splitting out image building from devtest_overcloud.sh
On Thu, 2015-04-16 at 02:59 +, Gregory Haynes wrote: Excerpts from Gregory Haynes's message of 2015-04-16 02:50:17 +: Excerpts from Dan Prince's message of 2015-04-15 02:14:12 +: I've been trying to cleanly model some Ceph and HA configurations in tripleo-ci that use Puppet (we are quite close to having these things in CI now!) Turns out the environment variables needed for these things are getting to be quite a mess. Furthermore we'd actually need to add to the environment variable madness to get it all working. And then there are optimization we'd like to add (like building a single image instead of one per role). One thing that would really help in this regard is splitting out image building from devtest_overcloud.sh. I took a stab at some initial patches to do this today. build-images: drive DIB via YAML config file https://review.openstack.org/#/c/173644/ devtest_overcloud.sh: split out image building https://review.openstack.org/#/c/173645/ If these sit well we could expand the effort to load images a bit more dynamically (a load-images script which could also be driven via a disk_images.yaml config file) and then I think devtest_overcloud.sh would be a lot more flexible for us Puppet users. Thoughts? I still have some questions myself but I wanted to get this out because we really do need some extra flexibility to be able to cleanly tune our scripts for more CI jobs. Dan Have you looked at possibly using infra's nodepool for this? It is a bit overkill, but currently nodepool lets you define a yaml file of images for it to build using dib. If were not ok with bringing in all the extras that nodepool has, maybe we could work on splitting out part of nodepool for our needs, and having both projects this. Cheers, Greg Did some digging and looks like infra has some planned work for this already[1]. This would be great for TripleO as well for the same reasons that infra wants it. Thanks for pointing this out Greg. I did speak w/ Monty on IRC and our tools do have things in common. The infra tool is here I believe: https://review.openstack.org/#/c/140840/12/tools/test-dib-upload.yaml And here is what I've currently got in TripleO: https://review.openstack.org/#/c/173644/6/scripts/build-images,cm For TripleO I was going more specifically for something that only enhanced DIB functionality, no external cloud references, etc. Although that could work too I think. Although I did overload things a bit by sharing the same config file with another tool called load-images so maybe we really are talking about the same things here :). At the end of the day this is how I'm trying to wire it into TripleO CI to support using a single image for Puppet: https://review.openstack.org/#/c/177060/1/toci_gate_test.sh,cm The goal being to save CI time mostly... but I think there are practical uses for having image building and loading abstracted from devtest across the board. Dan I do get that you have a need for this today though and what i'm describing is a ways out, so I am +1 on your current approach for now. Cheers, Greg 1: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] splitting out image building from devtest_overcloud.sh
Excerpts from Dan Prince's message of 2015-04-15 02:14:12 +: I've been trying to cleanly model some Ceph and HA configurations in tripleo-ci that use Puppet (we are quite close to having these things in CI now!) Turns out the environment variables needed for these things are getting to be quite a mess. Furthermore we'd actually need to add to the environment variable madness to get it all working. And then there are optimization we'd like to add (like building a single image instead of one per role). One thing that would really help in this regard is splitting out image building from devtest_overcloud.sh. I took a stab at some initial patches to do this today. build-images: drive DIB via YAML config file https://review.openstack.org/#/c/173644/ devtest_overcloud.sh: split out image building https://review.openstack.org/#/c/173645/ If these sit well we could expand the effort to load images a bit more dynamically (a load-images script which could also be driven via a disk_images.yaml config file) and then I think devtest_overcloud.sh would be a lot more flexible for us Puppet users. Thoughts? I still have some questions myself but I wanted to get this out because we really do need some extra flexibility to be able to cleanly tune our scripts for more CI jobs. Dan Have you looked at possibly using infra's nodepool for this? It is a bit overkill, but currently nodepool lets you define a yaml file of images for it to build using dib. If were not ok with bringing in all the extras that nodepool has, maybe we could work on splitting out part of nodepool for our needs, and having both projects this. Cheers, Greg __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] splitting out image building from devtest_overcloud.sh
Excerpts from Gregory Haynes's message of 2015-04-16 02:50:17 +: Excerpts from Dan Prince's message of 2015-04-15 02:14:12 +: I've been trying to cleanly model some Ceph and HA configurations in tripleo-ci that use Puppet (we are quite close to having these things in CI now!) Turns out the environment variables needed for these things are getting to be quite a mess. Furthermore we'd actually need to add to the environment variable madness to get it all working. And then there are optimization we'd like to add (like building a single image instead of one per role). One thing that would really help in this regard is splitting out image building from devtest_overcloud.sh. I took a stab at some initial patches to do this today. build-images: drive DIB via YAML config file https://review.openstack.org/#/c/173644/ devtest_overcloud.sh: split out image building https://review.openstack.org/#/c/173645/ If these sit well we could expand the effort to load images a bit more dynamically (a load-images script which could also be driven via a disk_images.yaml config file) and then I think devtest_overcloud.sh would be a lot more flexible for us Puppet users. Thoughts? I still have some questions myself but I wanted to get this out because we really do need some extra flexibility to be able to cleanly tune our scripts for more CI jobs. Dan Have you looked at possibly using infra's nodepool for this? It is a bit overkill, but currently nodepool lets you define a yaml file of images for it to build using dib. If were not ok with bringing in all the extras that nodepool has, maybe we could work on splitting out part of nodepool for our needs, and having both projects this. Cheers, Greg Did some digging and looks like infra has some planned work for this already[1]. This would be great for TripleO as well for the same reasons that infra wants it. I do get that you have a need for this today though and what i'm describing is a ways out, so I am +1 on your current approach for now. Cheers, Greg 1: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] splitting out image building from devtest_overcloud.sh
On 15/04/15 05:14, Dan Prince wrote: I've been trying to cleanly model some Ceph and HA configurations in tripleo-ci that use Puppet (we are quite close to having these things in CI now!) Turns out the environment variables needed for these things are getting to be quite a mess. Furthermore we'd actually need to add to the environment variable madness to get it all working. And then there are optimization we'd like to add (like building a single image instead of one per role). One thing that would really help in this regard is splitting out image building from devtest_overcloud.sh. I took a stab at some initial patches to do this today. build-images: drive DIB via YAML config file https://review.openstack.org/#/c/173644/ devtest_overcloud.sh: split out image building https://review.openstack.org/#/c/173645/ If these sit well we could expand the effort to load images a bit more dynamically (a load-images script which could also be driven via a disk_images.yaml config file) and then I think devtest_overcloud.sh would be a lot more flexible for us Puppet users. Thoughts? some initial thoughts on the review - imo you could even sell /#/c/173645/ as a nice tidy up regardless of the new build_images stuff (which looks excellent anyway) thanks, marios I still have some questions myself but I wanted to get this out because we really do need some extra flexibility to be able to cleanly tune our scripts for more CI jobs. Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] splitting out image building from devtest_overcloud.sh
I've been trying to cleanly model some Ceph and HA configurations in tripleo-ci that use Puppet (we are quite close to having these things in CI now!) Turns out the environment variables needed for these things are getting to be quite a mess. Furthermore we'd actually need to add to the environment variable madness to get it all working. And then there are optimization we'd like to add (like building a single image instead of one per role). One thing that would really help in this regard is splitting out image building from devtest_overcloud.sh. I took a stab at some initial patches to do this today. build-images: drive DIB via YAML config file https://review.openstack.org/#/c/173644/ devtest_overcloud.sh: split out image building https://review.openstack.org/#/c/173645/ If these sit well we could expand the effort to load images a bit more dynamically (a load-images script which could also be driven via a disk_images.yaml config file) and then I think devtest_overcloud.sh would be a lot more flexible for us Puppet users. Thoughts? I still have some questions myself but I wanted to get this out because we really do need some extra flexibility to be able to cleanly tune our scripts for more CI jobs. Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
