Re: [openstack-dev] [DIB] [Ironic] [TripleO] Moving IPA element out of DIB tree

[Jay Faulkner]

One more note I missed in the previous email.

On Jul 15, 2016, at 1:46 PM, Ben Nemec 
mailto:openst...@nemebean.com>> wrote:

I think this probably makes sense, but some more thoughts inline.

On 07/15/2016 03:13 PM, Stephane Miller wrote:
To better support diskimage-builder based IPA images going forward, we'd
like to move the ironic-agent element into the ironic-python-agent
repository. This will involve:

- Improving support for having multiple copies of an element, so that we
may deprecate the diskimage-builder repository copy of the element. See
this change and related: https://review.openstack.org/#/c/334785
- Moving the element into the repository. This change has been proposed
as https://review.openstack.org/#/c/335583/
- Deprecating the diskimage-builder copy of the element (TBD)
- Adding tests to gate IPA changes on DIB builds (TBD)

We could potentially add tripleo-ci to the IPA repo, which would take
care of this.  As an added bonus, it could cover both the introspection
and deployment use cases for IPA.

On the other hand, if a separate Ironic job were added to cover this,
tripleo could stop ever building new IPA images in CI except in the
promote jobs when we bump our version of IPA.  This would delay our
finding problems with IPA element changes, but realistically I'm not
sure how many of those are happening these days anyway.  I'd expect that
most changes are happening in IPA itself, which we don't currently CI.


We already have a well-established pattern for testing multiple ramdisks under 
IPA, and this "workflow" already basically works for DIB, however, is extremely 
awkward in terms of co-gating changes (such as needing to add dependencies to 
utilize new IPA features). Here's a basic breakdown:

We set IRONIC_BUILD_DEPLOY_RAMDISK to true, which instructs the Ironic devstack 
plugin to build a new ramdisk instead of downloading and using a prebuilt 
ramdisk from tarballs.openstack.org. Which 
ramdisk gets built is determined by IRONIC_RAMDISK_TYPE. For DIB, we already 
have all the code for a job to run this way (in fact; it'd be interesting to go 
ahead and add a non-voting version of this job). 
https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L1185

There's no need for new approaches to how IPA does CI to test this; the desire 
to have the DIB element in-tree is simply to allow parallelism with the other 
build methods (which maintain their build and dependencies in-tree) and to 
prevent a situation where IPA changes are blocked on DIB element changes 
merging into another repo. This is the same approach used for Ironic in 
devstack (plugins-in-tree) and that is being worked on for Ironic in tempest, 
and I don't want to repeat mistakes of CI past of having us blocked on merging 
packages on another project's core team. (Regardless of how friendly or 
responsive you are :D).

Thanks,
Jay Faulkner
OSIC

- Add upload of DIB-built images to 
tarballs.openstack.org
 (TBD)

We would also need to resolve https://review.openstack.org/#/c/334042/

I'm not clear why, but the ironic-agent element was given special
treatment in disk-image-create (which is evil, but what's done is done)
and we'd need to figure out why and a solution that wouldn't require
referencing an out-of-tree element in diskimage-builder.


Many IPA deployers currently use DIB based IPA images using the
ironic-agent element. However, IPA does not officially support DIB - IPA
changes are not tested against DIB, nor are DIB-built images published.

tripleo-ci actually does publish images, but they aren't well publicized
at this point, and it only does so when we promote a repo.


This has the following disadvantages:

- The DIB element is not versioned along with IPA, resulting in
potential version mismatch and breakage
- ironic-agent element changes are not tested with DIB prior to merge

This isn't true today.  tripleo-ci runs against all diskimage-builder
changes and uses an IPA ramdisk.  The version mismatch is a legit
problem with the current setup, although I'm not aware of any actual
breakages that have happened (which doesn't necessarily mean they
haven't :-).


Understandably, tripleo and other projects may have concerns with regard
to this change. I hope to start a discussion here so that those concerns
can be addressed. Further in-depth discussion of this issue can be found
in the relevant launchpad bug:
https://bugs.launchpad.net/ironic-python-agent/+bug/1590935

Thanks,
Stephane


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe

Re: [openstack-dev] [DIB] [Ironic] [TripleO] Moving IPA element out of DIB tree

[Jay Faulkner]

A few notes inline; I'm not the best technical contact for IPA+DIB integration, 
but going to add a little bit of context here so you all can well-understand 
our goals.

On Jul 15, 2016, at 4:28 PM, Gregory Haynes 
mailto:g...@greghaynes.net>> wrote:

On Fri, Jul 15, 2016, at 03:46 PM, Ben Nemec wrote:
I think this probably makes sense, but some more thoughts inline.

On 07/15/2016 03:13 PM, Stephane Miller wrote:
To better support diskimage-builder based IPA images going forward, we'd
like to move the ironic-agent element into the ironic-python-agent
repository. This will involve:

- Improving support for having multiple copies of an element, so that we
may deprecate the diskimage-builder repository copy of the element. See
this change and related: https://review.openstack.org/#/c/334785
- Moving the element into the repository. This change has been proposed
as https://review.openstack.org/#/c/335583/
- Deprecating the diskimage-builder copy of the element (TBD)
- Adding tests to gate IPA changes on DIB builds (TBD)

We now have some machinery to write per-element tests which result in an
image build and the ability to assert properties of that image. AFAIK no
downstreams of DIB have begun using it but this seems like a great
candidate.


We could potentially add tripleo-ci to the IPA repo, which would take
care of this.  As an added bonus, it could cover both the introspection
and deployment use cases for IPA.

On the other hand, if a separate Ironic job were added to cover this,
tripleo could stop ever building new IPA images in CI except in the
promote jobs when we bump our version of IPA.  This would delay our
finding problems with IPA element changes, but realistically I'm not
sure how many of those are happening these days anyway.  I'd expect that
most changes are happening in IPA itself, which we don't currently CI.

- Add upload of DIB-built images to 
tarballs.openstack.org
 (TBD)

We would also need to resolve https://review.openstack.org/#/c/334042/

I'm not clear why, but the ironic-agent element was given special
treatment in disk-image-create (which is evil, but what's done is done)
and we'd need to figure out why and a solution that wouldn't require
referencing an out-of-tree element in diskimage-builder.


I agree that this is something we should solve, but I don't think its a
blocker for the element moving out of tree - I think the (nasty) dib
special-casing will still apply as long as the element is named the
same?

I suspect this is a ways off, but an interesting question will be what
distros to base these images off of. AIUI the current published image is
CoreOS based which is something we haven't written an element for (yet).
I don't think there's any issues here, just a lot of options - Do we add
CoreOS support, do you publish multiple images built on various distros
we currently support?


Right now, IPA "officially" supports Tiny Core Linux and CoreOS images. We have 
people running DIB-built images based on ubuntu and fedora in production 
environments currently. The effort to get this work done for DIB is to get it 
gating on IPA changes and therefore "supported" as an official ramdisk.

Currently we publish IPA ramdisk images based on both TinyIPA (the name for our 
TCL image) and CoreOS. I'd imagine we'd end up having multiple supported DIB 
ramdisks, maybe one based on fedora, and one based on ubuntu/debian, since 
that's what the community has shown a liking for already.


Many IPA deployers currently use DIB based IPA images using the
ironic-agent element. However, IPA does not officially support DIB - IPA
changes are not tested against DIB, nor are DIB-built images published.

tripleo-ci actually does publish images, but they aren't well publicized
at this point, and it only does so when we promote a repo.


This has the following disadvantages:

- The DIB element is not versioned along with IPA, resulting in
potential version mismatch and breakage
- ironic-agent element changes are not tested with DIB prior to merge

This isn't true today.  tripleo-ci runs against all diskimage-builder
changes and uses an IPA ramdisk.  The version mismatch is a legit
problem with the current setup, although I'm not aware of any actual
breakages that have happened (which doesn't necessarily mean they
haven't :-).


I think there's another aspect to this which is that by hosting IPA in
tree we are effectively saying that DIB should co-gate with IPA changes
(how else can IPA test changes to its element?). The problem with this
is that DIB installs a lot of things and there isn't much value in us
co-gating with every thing we install - it also isn't sustainable.
Really, we want IPA to gate on changes to the IPA element and for DIB to
have robust enough testing that it will reliably produce a workable OS
for the IPA install logic to run in. I think that moving the IPA element
in to the IPA tree makes a lot of sense from this standpo

Re: [openstack-dev] [DIB] [Ironic] [TripleO] Moving IPA element out of DIB tree

[Gregory Haynes]

On Fri, Jul 15, 2016, at 03:46 PM, Ben Nemec wrote:
> I think this probably makes sense, but some more thoughts inline.
> 
> On 07/15/2016 03:13 PM, Stephane Miller wrote:
> > To better support diskimage-builder based IPA images going forward, we'd
> > like to move the ironic-agent element into the ironic-python-agent
> > repository. This will involve:
> > 
> > - Improving support for having multiple copies of an element, so that we
> > may deprecate the diskimage-builder repository copy of the element. See
> > this change and related: https://review.openstack.org/#/c/334785
> > - Moving the element into the repository. This change has been proposed
> > as https://review.openstack.org/#/c/335583/
> > - Deprecating the diskimage-builder copy of the element (TBD)
> > - Adding tests to gate IPA changes on DIB builds (TBD)

We now have some machinery to write per-element tests which result in an
image build and the ability to assert properties of that image. AFAIK no
downstreams of DIB have begun using it but this seems like a great
candidate.

> 
> We could potentially add tripleo-ci to the IPA repo, which would take
> care of this.  As an added bonus, it could cover both the introspection
> and deployment use cases for IPA.
> 
> On the other hand, if a separate Ironic job were added to cover this,
> tripleo could stop ever building new IPA images in CI except in the
> promote jobs when we bump our version of IPA.  This would delay our
> finding problems with IPA element changes, but realistically I'm not
> sure how many of those are happening these days anyway.  I'd expect that
> most changes are happening in IPA itself, which we don't currently CI.
> 
> > - Add upload of DIB-built images to tarballs.openstack.org
> >  (TBD)
> 
> We would also need to resolve https://review.openstack.org/#/c/334042/
> 
> I'm not clear why, but the ironic-agent element was given special
> treatment in disk-image-create (which is evil, but what's done is done)
> and we'd need to figure out why and a solution that wouldn't require
> referencing an out-of-tree element in diskimage-builder.
> 

I agree that this is something we should solve, but I don't think its a
blocker for the element moving out of tree - I think the (nasty) dib
special-casing will still apply as long as the element is named the
same?

I suspect this is a ways off, but an interesting question will be what
distros to base these images off of. AIUI the current published image is
CoreOS based which is something we haven't written an element for (yet).
I don't think there's any issues here, just a lot of options - Do we add
CoreOS support, do you publish multiple images built on various distros
we currently support?

> > 
> > Many IPA deployers currently use DIB based IPA images using the
> > ironic-agent element. However, IPA does not officially support DIB - IPA
> > changes are not tested against DIB, nor are DIB-built images published.
> 
> tripleo-ci actually does publish images, but they aren't well publicized
> at this point, and it only does so when we promote a repo.
> 
> > 
> > This has the following disadvantages:
> > 
> > - The DIB element is not versioned along with IPA, resulting in
> > potential version mismatch and breakage
> > - ironic-agent element changes are not tested with DIB prior to merge
> 
> This isn't true today.  tripleo-ci runs against all diskimage-builder
> changes and uses an IPA ramdisk.  The version mismatch is a legit
> problem with the current setup, although I'm not aware of any actual
> breakages that have happened (which doesn't necessarily mean they
> haven't :-).
> 

I think there's another aspect to this which is that by hosting IPA in
tree we are effectively saying that DIB should co-gate with IPA changes
(how else can IPA test changes to its element?). The problem with this
is that DIB installs a lot of things and there isn't much value in us
co-gating with every thing we install - it also isn't sustainable.
Really, we want IPA to gate on changes to the IPA element and for DIB to
have robust enough testing that it will reliably produce a workable OS
for the IPA install logic to run in. I think that moving the IPA element
in to the IPA tree makes a lot of sense from this standpoint.

As for breakages from not co-gating - all of the dib + ironic breakages
I remember were when we used the old ramdisk element which had a lot
more ironic specific logic in the element. Now that IPA is a thing and
isn't a bunch of bash inside of DIB the surface area for DIB to break
Ironic is actually pretty low (which is awesome).

> > 
> > Understandably, tripleo and other projects may have concerns with regard
> > to this change. I hope to start a discussion here so that those concerns
> > can be addressed. Further in-depth discussion of this issue can be found
> > in the relevant launchpad bug:
> > https://bugs.launchpad.net/ironic-python-agent/+bug/1590935
> > 
> > Thanks,
> > Stephane


Cheers,
Greg

___

Re: [openstack-dev] [DIB] [Ironic] [TripleO] Moving IPA element out of DIB tree

[Ben Nemec]

I think this probably makes sense, but some more thoughts inline.

On 07/15/2016 03:13 PM, Stephane Miller wrote:
> To better support diskimage-builder based IPA images going forward, we'd
> like to move the ironic-agent element into the ironic-python-agent
> repository. This will involve:
> 
> - Improving support for having multiple copies of an element, so that we
> may deprecate the diskimage-builder repository copy of the element. See
> this change and related: https://review.openstack.org/#/c/334785
> - Moving the element into the repository. This change has been proposed
> as https://review.openstack.org/#/c/335583/
> - Deprecating the diskimage-builder copy of the element (TBD)
> - Adding tests to gate IPA changes on DIB builds (TBD)

We could potentially add tripleo-ci to the IPA repo, which would take
care of this.  As an added bonus, it could cover both the introspection
and deployment use cases for IPA.

On the other hand, if a separate Ironic job were added to cover this,
tripleo could stop ever building new IPA images in CI except in the
promote jobs when we bump our version of IPA.  This would delay our
finding problems with IPA element changes, but realistically I'm not
sure how many of those are happening these days anyway.  I'd expect that
most changes are happening in IPA itself, which we don't currently CI.

> - Add upload of DIB-built images to tarballs.openstack.org
>  (TBD)

We would also need to resolve https://review.openstack.org/#/c/334042/

I'm not clear why, but the ironic-agent element was given special
treatment in disk-image-create (which is evil, but what's done is done)
and we'd need to figure out why and a solution that wouldn't require
referencing an out-of-tree element in diskimage-builder.

> 
> Many IPA deployers currently use DIB based IPA images using the
> ironic-agent element. However, IPA does not officially support DIB - IPA
> changes are not tested against DIB, nor are DIB-built images published.

tripleo-ci actually does publish images, but they aren't well publicized
at this point, and it only does so when we promote a repo.

> 
> This has the following disadvantages:
> 
> - The DIB element is not versioned along with IPA, resulting in
> potential version mismatch and breakage
> - ironic-agent element changes are not tested with DIB prior to merge

This isn't true today.  tripleo-ci runs against all diskimage-builder
changes and uses an IPA ramdisk.  The version mismatch is a legit
problem with the current setup, although I'm not aware of any actual
breakages that have happened (which doesn't necessarily mean they
haven't :-).

> 
> Understandably, tripleo and other projects may have concerns with regard
> to this change. I hope to start a discussion here so that those concerns
> can be addressed. Further in-depth discussion of this issue can be found
> in the relevant launchpad bug:
> https://bugs.launchpad.net/ironic-python-agent/+bug/1590935
> 
> Thanks,
> Stephane
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [DIB] [Ironic] [TripleO] Moving IPA element out of DIB tree

[Stephane Miller]

To better support diskimage-builder based IPA images going forward, we'd
like to move the ironic-agent element into the ironic-python-agent
repository. This will involve:

- Improving support for having multiple copies of an element, so that we
may deprecate the diskimage-builder repository copy of the element. See
this change and related: https://review.openstack.org/#/c/334785
- Moving the element into the repository. This change has been proposed as
https://review.openstack.org/#/c/335583/
- Deprecating the diskimage-builder copy of the element (TBD)
- Adding tests to gate IPA changes on DIB builds (TBD)
- Add upload of DIB-built images to tarballs.openstack.org (TBD)

Many IPA deployers currently use DIB based IPA images using the
ironic-agent element. However, IPA does not officially support DIB - IPA
changes are not tested against DIB, nor are DIB-built images published.

This has the following disadvantages:

- The DIB element is not versioned along with IPA, resulting in potential
version mismatch and breakage
- ironic-agent element changes are not tested with DIB prior to merge

Understandably, tripleo and other projects may have concerns with regard to
this change. I hope to start a discussion here so that those concerns can
be addressed. Further in-depth discussion of this issue can be found in the
relevant launchpad bug:
https://bugs.launchpad.net/ironic-python-agent/+bug/1590935

Thanks,
Stephane
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev