Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
On Tue, Jan 13, 2015 at 09:12:59AM +1300, Steve Baker wrote: > On 09/01/15 07:06, Gregory Haynes wrote: > >Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +: > >>Hi all, > >> > >>I'm trying to test a fedora-software-config image with some updated > >>components. I need: > >> > >>- Install latest master os-apply-config (the commit I want isn't released) > >>- Install os-refresh-config fork from > >>https://review.openstack.org/#/c/145764 > >> > >>I can't even get the o-a-c from master part working: > >> > >>export PATH="${PWD}/dib-utils/bin:$PATH" > >>export > >>ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements > >>export DIB_INSTALLTYPE_os_apply_config=source > >> > >>diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ > >> os-collect-config os-refresh-config os-apply-config \ > >> heat-config-ansible \ > >> heat-config-cfn-init \ > >> heat-config-docker \ > >> heat-config-puppet \ > >> heat-config-salt \ > >> heat-config-script \ > >> ntp \ > >> -o fedora-software-config.qcow2 > >> > >>This is what I'm doing, both tools end up as pip installed versions AFAICS, > >>so I've had to resort to manually hacking the image post-DiB using > >>virt-copy-in. > >> > >>Pretty sure there's a way to make DiB do this, but don't know what, anyone > >>able to share some clues? Do I have to hack the elements, or is there a > >>better way? > >> > >>The docs are pretty sparse, so any help would be much appreciated! :) > >> > >>Thanks, > >> > >>Steve > >> > >Hey Steve, > > > >source-repositories is your friend here :) (check out > >dib/elements/source-repositires/README). One potential gotcha is that > >because source-repositires is an element it really only applies to tools > >used within images (and os-apply-config is used outside the image). To > >fix this we have a shim in tripleo-incubator/scripts/pull-tools which > >emulates the functionality of source-repositories. > > > >Example usage: > > > >* checkout os-apply-config to the ref you wish to use > >* export DIB_REPOLOCATION_os_apply_config="/path/to/oac" > >* export DIB_REPOREF_os_refresh_config="refs/changes/64/145764/1" > >* start your devtesting > > > > > The good news is that devstack is already set up to do this. When > HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the currently > checked-out os-*-config tools, build a pip repo and configure apache to > serve it. > > Then the elements *should* install from these packages - we're not gating on > this functionality (yet) so its possible it has regressed but shouldn't be > too hard to get going again. Thanks for this - after a small fix, it works great! :) https://review.openstack.org/147109 It'd be good if we could expose image_elements as a variable, and provide a little script which enables easy rebuilding of the image without re-stacking. I'll probably take a look at both when I get time. Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
> From: Steve Baker > To: openstack-dev@lists.openstack.org > Date: 12/01/2015 21:16 > Subject: Re: [openstack-dev] [heat][tripleo] Making diskimage- > builder install from forked repo? > > On 09/01/15 07:06, Gregory Haynes wrote: > > Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +: > >> Hi all, > >> > >> I'm trying to test a fedora-software-config image with some updated > >> components. I need: > >> > >> - Install latest master os-apply-config (the commit I want isn't released) > >> - Install os-refresh-config fork from https:// > review.openstack.org/#/c/145764 > >> > >> I can't even get the o-a-c from master part working: > >> > >> export PATH="${PWD}/dib-utils/bin:$PATH" > >> export > >> ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/ > software-config/elements > >> export DIB_INSTALLTYPE_os_apply_config=source > >> > >> diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ > >>os-collect-config os-refresh-config os-apply-config \ > >>heat-config-ansible \ > >>heat-config-cfn-init \ > >>heat-config-docker \ > >>heat-config-puppet \ > >>heat-config-salt \ > >>heat-config-script \ > >>ntp \ > >>-o fedora-software-config.qcow2 > >> > >> This is what I'm doing, both tools end up as pip installed versions AFAICS, > >> so I've had to resort to manually hacking the image post-DiB using > >> virt-copy-in. > >> > >> Pretty sure there's a way to make DiB do this, but don't know what, anyone > >> able to share some clues? Do I have to hack the elements, or is there a > >> better way? > >> > >> The docs are pretty sparse, so any help would be much appreciated! :) > >> > >> Thanks, > >> > >> Steve > >> > > Hey Steve, > > > > source-repositories is your friend here :) (check out > > dib/elements/source-repositires/README). One potential gotcha is that > > because source-repositires is an element it really only applies to tools > > used within images (and os-apply-config is used outside the image). To > > fix this we have a shim in tripleo-incubator/scripts/pull-tools which > > emulates the functionality of source-repositories. > > > > Example usage: > > > > * checkout os-apply-config to the ref you wish to use > > * export DIB_REPOLOCATION_os_apply_config="/path/to/oac" > > * export DIB_REPOREF_os_refresh_config="refs/changes/64/145764/1" > > * start your devtesting > > > > > The good news is that devstack is already set up to do this. When > HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the > currently checked-out os-*-config tools, build a pip repo and configure > apache to serve it. Very cool! Didn't know about it. Thanks for sharing this information :-) > > Then the elements *should* install from these packages - we're not > gating on this functionality (yet) so its possible it has regressed but > shouldn't be too hard to get going again. > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
On 09/01/15 07:06, Gregory Haynes wrote: Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +: Hi all, I'm trying to test a fedora-software-config image with some updated components. I need: - Install latest master os-apply-config (the commit I want isn't released) - Install os-refresh-config fork from https://review.openstack.org/#/c/145764 I can't even get the o-a-c from master part working: export PATH="${PWD}/dib-utils/bin:$PATH" export ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements export DIB_INSTALLTYPE_os_apply_config=source diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ os-collect-config os-refresh-config os-apply-config \ heat-config-ansible \ heat-config-cfn-init \ heat-config-docker \ heat-config-puppet \ heat-config-salt \ heat-config-script \ ntp \ -o fedora-software-config.qcow2 This is what I'm doing, both tools end up as pip installed versions AFAICS, so I've had to resort to manually hacking the image post-DiB using virt-copy-in. Pretty sure there's a way to make DiB do this, but don't know what, anyone able to share some clues? Do I have to hack the elements, or is there a better way? The docs are pretty sparse, so any help would be much appreciated! :) Thanks, Steve Hey Steve, source-repositories is your friend here :) (check out dib/elements/source-repositires/README). One potential gotcha is that because source-repositires is an element it really only applies to tools used within images (and os-apply-config is used outside the image). To fix this we have a shim in tripleo-incubator/scripts/pull-tools which emulates the functionality of source-repositories. Example usage: * checkout os-apply-config to the ref you wish to use * export DIB_REPOLOCATION_os_apply_config="/path/to/oac" * export DIB_REPOREF_os_refresh_config="refs/changes/64/145764/1" * start your devtesting The good news is that devstack is already set up to do this. When HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the currently checked-out os-*-config tools, build a pip repo and configure apache to serve it. Then the elements *should* install from these packages - we're not gating on this functionality (yet) so its possible it has regressed but shouldn't be too hard to get going again. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Excerpts from Chris Jones's message of 2015-01-08 10:16:14 -0800: > Hi > > > On 8 Jan 2015, at 17:58, Clint Byrum wrote: > > > > Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800: > > So you can probably setup a devpi instance locally, and upload the > > commits you want to it, and then build the image with the 'pypi' element > > Given that we have a pretty good release frequency of all our tools, is this > burden on devs/testers actually justified at this point, versus the potential > consistency we could have with source repo flexibility in other openstack > components? > We've been discussing in #tripleo and I think you're right. I think we can solve this by just switching to source-repositories and providing a relatively simple tool to set release tags when people want released versions only. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Excerpts from Gregory Haynes's message of 2015-01-08 18:06:16 +: > Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +: > > Hi all, > > > > I'm trying to test a fedora-software-config image with some updated > > components. I need: > > > > - Install latest master os-apply-config (the commit I want isn't released) > > - Install os-refresh-config fork from > > https://review.openstack.org/#/c/145764 > > > > I can't even get the o-a-c from master part working: > > > > export PATH="${PWD}/dib-utils/bin:$PATH" > > export > > ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements > > export DIB_INSTALLTYPE_os_apply_config=source > > > > diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ > > os-collect-config os-refresh-config os-apply-config \ > > heat-config-ansible \ > > heat-config-cfn-init \ > > heat-config-docker \ > > heat-config-puppet \ > > heat-config-salt \ > > heat-config-script \ > > ntp \ > > -o fedora-software-config.qcow2 > > > > This is what I'm doing, both tools end up as pip installed versions AFAICS, > > so I've had to resort to manually hacking the image post-DiB using > > virt-copy-in. > > > > Pretty sure there's a way to make DiB do this, but don't know what, anyone > > able to share some clues? Do I have to hack the elements, or is there a > > better way? > > > > The docs are pretty sparse, so any help would be much appreciated! :) > > > > Thanks, > > > > Steve > > > > Hey Steve, > > source-repositories is your friend here :) (check out > dib/elements/source-repositires/README). One potential gotcha is that > because source-repositires is an element it really only applies to tools > used within images (and os-apply-config is used outside the image). To > fix this we have a shim in tripleo-incubator/scripts/pull-tools which > emulates the functionality of source-repositories. > > Example usage: > > * checkout os-apply-config to the ref you wish to use > * export DIB_REPOLOCATION_os_apply_config="/path/to/oac" > * export DIB_REPOREF_os_refresh_config="refs/changes/64/145764/1" > * start your devtesting Actually, Chris's response is 100% correct. Even in the source installtype we appear to be pip installing these tools so this will not work. In our CI we work around this by creating a local pypi mirror and configuring pip to fall back to an upstream mirror. We then build sdist's for anything we want to install via git and add them to our 'overlay mirror': Code: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_devtest.sh#n139 Obviously, this isnt the most user friendly approach, but its an option. Good luck, Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Hi > On 8 Jan 2015, at 17:58, Clint Byrum wrote: > > Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800: > So you can probably setup a devpi instance locally, and upload the > commits you want to it, and then build the image with the 'pypi' element Given that we have a pretty good release frequency of all our tools, is this burden on devs/testers actually justified at this point, versus the potential consistency we could have with source repo flexibility in other openstack components? Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +: > Hi all, > > I'm trying to test a fedora-software-config image with some updated > components. I need: > > - Install latest master os-apply-config (the commit I want isn't released) > - Install os-refresh-config fork from https://review.openstack.org/#/c/145764 > > I can't even get the o-a-c from master part working: > > export PATH="${PWD}/dib-utils/bin:$PATH" > export > ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements > export DIB_INSTALLTYPE_os_apply_config=source > > diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ > os-collect-config os-refresh-config os-apply-config \ > heat-config-ansible \ > heat-config-cfn-init \ > heat-config-docker \ > heat-config-puppet \ > heat-config-salt \ > heat-config-script \ > ntp \ > -o fedora-software-config.qcow2 > > This is what I'm doing, both tools end up as pip installed versions AFAICS, > so I've had to resort to manually hacking the image post-DiB using > virt-copy-in. > > Pretty sure there's a way to make DiB do this, but don't know what, anyone > able to share some clues? Do I have to hack the elements, or is there a > better way? > > The docs are pretty sparse, so any help would be much appreciated! :) > > Thanks, > > Steve > Hey Steve, source-repositories is your friend here :) (check out dib/elements/source-repositires/README). One potential gotcha is that because source-repositires is an element it really only applies to tools used within images (and os-apply-config is used outside the image). To fix this we have a shim in tripleo-incubator/scripts/pull-tools which emulates the functionality of source-repositories. Example usage: * checkout os-apply-config to the ref you wish to use * export DIB_REPOLOCATION_os_apply_config="/path/to/oac" * export DIB_REPOREF_os_refresh_config="refs/changes/64/145764/1" * start your devtesting HTH, Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800: > Hi all, > > I'm trying to test a fedora-software-config image with some updated > components. I need: > > - Install latest master os-apply-config (the commit I want isn't released) > - Install os-refresh-config fork from https://review.openstack.org/#/c/145764 > > I can't even get the o-a-c from master part working: > > export PATH="${PWD}/dib-utils/bin:$PATH" > export > ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements > export DIB_INSTALLTYPE_os_apply_config=source > > diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ > os-collect-config os-refresh-config os-apply-config \ > heat-config-ansible \ > heat-config-cfn-init \ > heat-config-docker \ > heat-config-puppet \ > heat-config-salt \ > heat-config-script \ > ntp \ > -o fedora-software-config.qcow2 > > This is what I'm doing, both tools end up as pip installed versions AFAICS, > so I've had to resort to manually hacking the image post-DiB using > virt-copy-in. > > Pretty sure there's a way to make DiB do this, but don't know what, anyone > able to share some clues? Do I have to hack the elements, or is there a > better way? > > The docs are pretty sparse, so any help would be much appreciated! :) > Hi Steve. The os-*-config tools represent a bit of a quandry for us, as we want to test and run with released versions, not latest git, so the elements just install from pypi. I believe we use devpi in testing to test new commits to the tools themselves. So you can probably setup a devpi instance locally, and upload the commits you want to it, and then build the image with the 'pypi' element added and this: PYPI_MIRROR_URL=http://localhost:3141/ See diskimage-builder/elements/pypi/README.md for more info on how to set this up. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Hi > On 8 Jan 2015, at 17:37, Steven Hardy wrote: > Pretty sure there's a way to make DiB do this, but don't know what, anyone > able to share some clues? Do I have to hack the elements, or is there a > better way? > > The docs are pretty sparse, so any help would be much appreciated! :) We do have a mechanism for overriding the git sources for things, but os-*-config don't use them at the moment, they either install from packages or pip. I'm not sure what the rationale was for not including a git source for those tools, but I think we should do it, even if it's limited to situations where the procedure for overriding sources is being followed. (The procedure that should be used is the DIB_REPO* environment variables documented in diskimage-builder/elements/source-repositories/README.md) So, for now I think you're going to be stuck hacking the elements, unfortunately. Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?
Hi all, I'm trying to test a fedora-software-config image with some updated components. I need: - Install latest master os-apply-config (the commit I want isn't released) - Install os-refresh-config fork from https://review.openstack.org/#/c/145764 I can't even get the o-a-c from master part working: export PATH="${PWD}/dib-utils/bin:$PATH" export ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements export DIB_INSTALLTYPE_os_apply_config=source diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \ os-collect-config os-refresh-config os-apply-config \ heat-config-ansible \ heat-config-cfn-init \ heat-config-docker \ heat-config-puppet \ heat-config-salt \ heat-config-script \ ntp \ -o fedora-software-config.qcow2 This is what I'm doing, both tools end up as pip installed versions AFAICS, so I've had to resort to manually hacking the image post-DiB using virt-copy-in. Pretty sure there's a way to make DiB do this, but don't know what, anyone able to share some clues? Do I have to hack the elements, or is there a better way? The docs are pretty sparse, so any help would be much appreciated! :) Thanks, Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev