Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-14 Thread Steven Hardy 
On Tue, Jan 13, 2015 at 09:12:59AM +1300, Steve Baker wrote:
 On 09/01/15 07:06, Gregory Haynes wrote:
 Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +:
 Hi all,
 
 I'm trying to test a fedora-software-config image with some updated
 components.  I need:
 
 - Install latest master os-apply-config (the commit I want isn't released)
 - Install os-refresh-config fork from 
 https://review.openstack.org/#/c/145764
 
 I can't even get the o-a-c from master part working:
 
 export PATH=${PWD}/dib-utils/bin:$PATH
 export
 ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
 export DIB_INSTALLTYPE_os_apply_config=source
 
 diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
os-collect-config os-refresh-config os-apply-config \
heat-config-ansible \
heat-config-cfn-init \
heat-config-docker \
heat-config-puppet \
heat-config-salt \
heat-config-script \
ntp \
-o fedora-software-config.qcow2
 
 This is what I'm doing, both tools end up as pip installed versions AFAICS,
 so I've had to resort to manually hacking the image post-DiB using
 virt-copy-in.
 
 Pretty sure there's a way to make DiB do this, but don't know what, anyone
 able to share some clues?  Do I have to hack the elements, or is there a
 better way?
 
 The docs are pretty sparse, so any help would be much appreciated! :)
 
 Thanks,
 
 Steve
 
 Hey Steve,
 
 source-repositories is your friend here :) (check out
 dib/elements/source-repositires/README). One potential gotcha is that
 because source-repositires is an element it really only applies to tools
 used within images (and os-apply-config is used outside the image). To
 fix this we have a shim in tripleo-incubator/scripts/pull-tools which
 emulates the functionality of source-repositories.
 
 Example usage:
 
 * checkout os-apply-config to the ref you wish to use
 * export DIB_REPOLOCATION_os_apply_config=/path/to/oac
 * export DIB_REPOREF_os_refresh_config=refs/changes/64/145764/1
 * start your devtesting
 
 
 The good news is that devstack is already set up to do this. When
 HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the currently
 checked-out os-*-config tools, build a pip repo and configure apache to
 serve it.
 
 Then the elements *should* install from these packages - we're not gating on
 this functionality (yet) so its possible it has regressed but shouldn't be
 too hard to get going again.

Thanks for this - after a small fix, it works great! :)

https://review.openstack.org/147109

It'd be good if we could expose image_elements as a variable, and provide a
little script which enables easy rebuilding of the image without
re-stacking.  I'll probably take a look at both when I get time.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-13 Thread Thomas Spatzier 
 From: Steve Baker sba...@redhat.com
 To: openstack-dev@lists.openstack.org
 Date: 12/01/2015 21:16
 Subject: Re: [openstack-dev] [heat][tripleo] Making diskimage-
 builder install from forked repo?

 On 09/01/15 07:06, Gregory Haynes wrote:
  Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +:
  Hi all,
 
  I'm trying to test a fedora-software-config image with some updated
  components.  I need:
 
  - Install latest master os-apply-config (the commit I want isn't
released)
  - Install os-refresh-config fork from https://
 review.openstack.org/#/c/145764
 
  I can't even get the o-a-c from master part working:
 
  export PATH=${PWD}/dib-utils/bin:$PATH
  export
  ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/
 software-config/elements
  export DIB_INSTALLTYPE_os_apply_config=source
 
  diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
 os-collect-config os-refresh-config os-apply-config \
 heat-config-ansible \
 heat-config-cfn-init \
 heat-config-docker \
 heat-config-puppet \
 heat-config-salt \
 heat-config-script \
 ntp \
 -o fedora-software-config.qcow2
 
  This is what I'm doing, both tools end up as pip installed versions
AFAICS,
  so I've had to resort to manually hacking the image post-DiB using
  virt-copy-in.
 
  Pretty sure there's a way to make DiB do this, but don't know what,
anyone
  able to share some clues?  Do I have to hack the elements, or is there
a
  better way?
 
  The docs are pretty sparse, so any help would be much appreciated! :)
 
  Thanks,
 
  Steve
 
  Hey Steve,
 
  source-repositories is your friend here :) (check out
  dib/elements/source-repositires/README). One potential gotcha is that
  because source-repositires is an element it really only applies to
tools
  used within images (and os-apply-config is used outside the image). To
  fix this we have a shim in tripleo-incubator/scripts/pull-tools which
  emulates the functionality of source-repositories.
 
  Example usage:
 
  * checkout os-apply-config to the ref you wish to use
  * export DIB_REPOLOCATION_os_apply_config=/path/to/oac
  * export DIB_REPOREF_os_refresh_config=refs/changes/64/145764/1
  * start your devtesting
 
 
 The good news is that devstack is already set up to do this. When
 HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the
 currently checked-out os-*-config tools, build a pip repo and configure
 apache to serve it.

Very cool! Didn't know about it. Thanks for sharing this information :-)


 Then the elements *should* install from these packages - we're not
 gating on this functionality (yet) so its possible it has regressed but
 shouldn't be too hard to get going again.




__
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-12 Thread Steve Baker 

On 09/01/15 07:06, Gregory Haynes wrote:

Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +:

Hi all,

I'm trying to test a fedora-software-config image with some updated
components.  I need:

- Install latest master os-apply-config (the commit I want isn't released)
- Install os-refresh-config fork from https://review.openstack.org/#/c/145764

I can't even get the o-a-c from master part working:

export PATH=${PWD}/dib-utils/bin:$PATH
export
ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
export DIB_INSTALLTYPE_os_apply_config=source

diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
   os-collect-config os-refresh-config os-apply-config \
   heat-config-ansible \
   heat-config-cfn-init \
   heat-config-docker \
   heat-config-puppet \
   heat-config-salt \
   heat-config-script \
   ntp \
   -o fedora-software-config.qcow2

This is what I'm doing, both tools end up as pip installed versions AFAICS,
so I've had to resort to manually hacking the image post-DiB using
virt-copy-in.

Pretty sure there's a way to make DiB do this, but don't know what, anyone
able to share some clues?  Do I have to hack the elements, or is there a
better way?

The docs are pretty sparse, so any help would be much appreciated! :)

Thanks,

Steve


Hey Steve,

source-repositories is your friend here :) (check out
dib/elements/source-repositires/README). One potential gotcha is that
because source-repositires is an element it really only applies to tools
used within images (and os-apply-config is used outside the image). To
fix this we have a shim in tripleo-incubator/scripts/pull-tools which
emulates the functionality of source-repositories.

Example usage:

* checkout os-apply-config to the ref you wish to use
* export DIB_REPOLOCATION_os_apply_config=/path/to/oac
* export DIB_REPOREF_os_refresh_config=refs/changes/64/145764/1
* start your devtesting


The good news is that devstack is already set up to do this. When 
HEAT_CREATE_TEST_IMAGE=True devstack will build packages from the 
currently checked-out os-*-config tools, build a pip repo and configure 
apache to serve it.


Then the elements *should* install from these packages - we're not 
gating on this functionality (yet) so its possible it has regressed but 
shouldn't be too hard to get going again.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Steven Hardy 
Hi all,

I'm trying to test a fedora-software-config image with some updated
components.  I need:

- Install latest master os-apply-config (the commit I want isn't released)
- Install os-refresh-config fork from https://review.openstack.org/#/c/145764

I can't even get the o-a-c from master part working:

export PATH=${PWD}/dib-utils/bin:$PATH
export
ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
export DIB_INSTALLTYPE_os_apply_config=source

diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
  os-collect-config os-refresh-config os-apply-config \
  heat-config-ansible \
  heat-config-cfn-init \
  heat-config-docker \
  heat-config-puppet \
  heat-config-salt \
  heat-config-script \
  ntp \
  -o fedora-software-config.qcow2

This is what I'm doing, both tools end up as pip installed versions AFAICS,
so I've had to resort to manually hacking the image post-DiB using
virt-copy-in.

Pretty sure there's a way to make DiB do this, but don't know what, anyone
able to share some clues?  Do I have to hack the elements, or is there a
better way?

The docs are pretty sparse, so any help would be much appreciated! :)

Thanks,

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Chris Jones 
Hi

 On 8 Jan 2015, at 17:37, Steven Hardy sha...@redhat.com wrote:
 Pretty sure there's a way to make DiB do this, but don't know what, anyone
 able to share some clues?  Do I have to hack the elements, or is there a
 better way?
 
 The docs are pretty sparse, so any help would be much appreciated! :)


We do have a mechanism for overriding the git sources for things, but 
os-*-config don't use them at the moment, they either install from packages or 
pip. I'm not sure what the rationale was for not including a git source for 
those tools, but I think we should do it, even if it's limited to situations 
where the procedure for overriding sources is being followed.

(The procedure that should be used is the DIB_REPO* environment variables 
documented in diskimage-builder/elements/source-repositories/README.md)

So, for now I think you're going to be stuck hacking the elements, 
unfortunately.

Cheers,

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Clint Byrum 
Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800:
 Hi all,
 
 I'm trying to test a fedora-software-config image with some updated
 components.  I need:
 
 - Install latest master os-apply-config (the commit I want isn't released)
 - Install os-refresh-config fork from https://review.openstack.org/#/c/145764
 
 I can't even get the o-a-c from master part working:
 
 export PATH=${PWD}/dib-utils/bin:$PATH
 export
 ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
 export DIB_INSTALLTYPE_os_apply_config=source
 
 diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
   os-collect-config os-refresh-config os-apply-config \
   heat-config-ansible \
   heat-config-cfn-init \
   heat-config-docker \
   heat-config-puppet \
   heat-config-salt \
   heat-config-script \
   ntp \
   -o fedora-software-config.qcow2
 
 This is what I'm doing, both tools end up as pip installed versions AFAICS,
 so I've had to resort to manually hacking the image post-DiB using
 virt-copy-in.
 
 Pretty sure there's a way to make DiB do this, but don't know what, anyone
 able to share some clues?  Do I have to hack the elements, or is there a
 better way?
 
 The docs are pretty sparse, so any help would be much appreciated! :)
 

Hi Steve. The os-*-config tools represent a bit of a quandry for us,
as we want to test and run with released versions, not latest git, so
the elements just install from pypi. I believe we use devpi in testing
to test new commits to the tools themselves.

So you can probably setup a devpi instance locally, and upload the
commits you want to it, and then build the image with the 'pypi' element
added and this:

PYPI_MIRROR_URL=http://localhost:3141/

See diskimage-builder/elements/pypi/README.md for more info on how to
set this up.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Gregory Haynes 
Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +:
 Hi all,
 
 I'm trying to test a fedora-software-config image with some updated
 components.  I need:
 
 - Install latest master os-apply-config (the commit I want isn't released)
 - Install os-refresh-config fork from https://review.openstack.org/#/c/145764
 
 I can't even get the o-a-c from master part working:
 
 export PATH=${PWD}/dib-utils/bin:$PATH
 export
 ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
 export DIB_INSTALLTYPE_os_apply_config=source
 
 diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
   os-collect-config os-refresh-config os-apply-config \
   heat-config-ansible \
   heat-config-cfn-init \
   heat-config-docker \
   heat-config-puppet \
   heat-config-salt \
   heat-config-script \
   ntp \
   -o fedora-software-config.qcow2
 
 This is what I'm doing, both tools end up as pip installed versions AFAICS,
 so I've had to resort to manually hacking the image post-DiB using
 virt-copy-in.
 
 Pretty sure there's a way to make DiB do this, but don't know what, anyone
 able to share some clues?  Do I have to hack the elements, or is there a
 better way?
 
 The docs are pretty sparse, so any help would be much appreciated! :)
 
 Thanks,
 
 Steve
 

Hey Steve,

source-repositories is your friend here :) (check out
dib/elements/source-repositires/README). One potential gotcha is that
because source-repositires is an element it really only applies to tools
used within images (and os-apply-config is used outside the image). To
fix this we have a shim in tripleo-incubator/scripts/pull-tools which
emulates the functionality of source-repositories.

Example usage:

* checkout os-apply-config to the ref you wish to use
* export DIB_REPOLOCATION_os_apply_config=/path/to/oac
* export DIB_REPOREF_os_refresh_config=refs/changes/64/145764/1
* start your devtesting

HTH,
Greg

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Gregory Haynes 
Excerpts from Gregory Haynes's message of 2015-01-08 18:06:16 +:
 Excerpts from Steven Hardy's message of 2015-01-08 17:37:55 +:
  Hi all,
  
  I'm trying to test a fedora-software-config image with some updated
  components.  I need:
  
  - Install latest master os-apply-config (the commit I want isn't released)
  - Install os-refresh-config fork from 
  https://review.openstack.org/#/c/145764
  
  I can't even get the o-a-c from master part working:
  
  export PATH=${PWD}/dib-utils/bin:$PATH
  export
  ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements
  export DIB_INSTALLTYPE_os_apply_config=source
  
  diskimage-builder/bin/disk-image-create vm fedora selinux-permissive \
os-collect-config os-refresh-config os-apply-config \
heat-config-ansible \
heat-config-cfn-init \
heat-config-docker \
heat-config-puppet \
heat-config-salt \
heat-config-script \
ntp \
-o fedora-software-config.qcow2
  
  This is what I'm doing, both tools end up as pip installed versions AFAICS,
  so I've had to resort to manually hacking the image post-DiB using
  virt-copy-in.
  
  Pretty sure there's a way to make DiB do this, but don't know what, anyone
  able to share some clues?  Do I have to hack the elements, or is there a
  better way?
  
  The docs are pretty sparse, so any help would be much appreciated! :)
  
  Thanks,
  
  Steve
  
 
 Hey Steve,
 
 source-repositories is your friend here :) (check out
 dib/elements/source-repositires/README). One potential gotcha is that
 because source-repositires is an element it really only applies to tools
 used within images (and os-apply-config is used outside the image). To
 fix this we have a shim in tripleo-incubator/scripts/pull-tools which
 emulates the functionality of source-repositories.
 
 Example usage:
 
 * checkout os-apply-config to the ref you wish to use
 * export DIB_REPOLOCATION_os_apply_config=/path/to/oac
 * export DIB_REPOREF_os_refresh_config=refs/changes/64/145764/1
 * start your devtesting

Actually, Chris's response is 100% correct. Even in the source
installtype we appear to be pip installing these tools so this will not
work.

In our CI we work around this by creating a local pypi mirror and
configuring pip to fall back to an upstream mirror. We then build
sdist's for anything we want to install via git and add them to our
'overlay mirror':

Code:
http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_devtest.sh#n139

Obviously, this isnt the most user friendly approach, but its an option.

Good luck,
Greg

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Chris Jones 
Hi

 On 8 Jan 2015, at 17:58, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800:
 So you can probably setup a devpi instance locally, and upload the
 commits you want to it, and then build the image with the 'pypi' element

Given that we have a pretty good release frequency of all our tools, is this 
burden on devs/testers actually justified at this point, versus the potential 
consistency we could have with source repo flexibility in other openstack 
components?

Cheers,

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Making diskimage-builder install from forked repo?

2015-01-08 Thread Clint Byrum 
Excerpts from Chris Jones's message of 2015-01-08 10:16:14 -0800:
 Hi
 
  On 8 Jan 2015, at 17:58, Clint Byrum cl...@fewbar.com wrote:
  
  Excerpts from Steven Hardy's message of 2015-01-08 09:37:55 -0800:
  So you can probably setup a devpi instance locally, and upload the
  commits you want to it, and then build the image with the 'pypi' element
 
 Given that we have a pretty good release frequency of all our tools, is this 
 burden on devs/testers actually justified at this point, versus the potential 
 consistency we could have with source repo flexibility in other openstack 
 components?
 

We've been discussing in #tripleo and I think you're right. I think we
can solve this by just switching to source-repositories and providing
a relatively simple tool to set release tags when people want released
versions only.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev