Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-17 Thread Ian Wienand
On 03/17/2017 03:46 AM, Steven Hardy wrote:
> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch
> (undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch

/bin is a link to /usr/bin?  So I think this is the same and this is
the dib-run-parts as pacakged by dib-utils.

> (undercloud) [stack@undercloud ~]$ rpm -qf 
> /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
> diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch

This is dib's "private" copy.  As I mentioned in the other mail, the
intention was to vendor this so we could re-write for dib-specific
needs if need be (given future requirements such as running in
restricted container environments).  I believe having dib exporting
this was an (my) oversight.  I have proposed [1] to remove this.

> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
> file /usr/local/bin/dib-run-parts is not owned by any package

This would come from the image build process.  We copy dib-run-parts
into the chroot to run in-target scripts [2] but we never actually
remove it.  This seems to me to also be a bug and I've proposed [3] to
run this out of /tmp and clean it up.

> But the exciting thing from a rolling-out-bugfixes perspective is that the
> one actually running via o-r-c isn't either of the packaged versions (doh!)
> so we probably need to track down which element is installing it.
>
> This is a little OT for this thread (sorry), but hopefully provides more
> context around my concerns about creating another fork etc.

I don't want us to get a little too "left-pad" [4] with this 95ish
lines of shell :) I think this stack clears things up.

tl;dr

 - dib version should be vendored; not in path & not exported [1]
 - unnecessary /usr/local/bin version removed [3]
 - dib-utils provides /usr/bin/ version

Cross-ports between the vendored dib version and dib-utils should be
trivial given what it is.  If dib wants to rewrite it's vendored
version, or remove it completely, this will not affect anyone
depending on dib-utils.

Thanks,

-i

[1] https://review.openstack.org/446285 (dib: do not provide dib-run-parts)
[2] 
https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dib-run-parts/root.d/90-base-dib-run-parts
[3] https://review.openstack.org/446769 (dib: run chroot dib-run-parts out of 
/tmp)
[4] http://left-pad.io/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-16 Thread Clint Byrum
Excerpts from Clark Boylan's message of 2017-03-16 10:16:37 -0700:
> On Thu, Mar 16, 2017, at 09:46 AM, Steven Hardy wrote:
> > On Thu, Mar 16, 2017 at 10:30:48AM -0500, Gregory Haynes wrote:
> > > On Thu, Mar 16, 2017, at 05:18 AM, Steven Hardy wrote:
> > > > On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> > > > > While looking through the dib v2 changes after the feature branch was 
> > > > > merged
> > > > > to master, I noticed this commit[1], which bring dib-run-parts back 
> > > > > into dib
> > > > > itself.  Unfortunately I missed the original proposal to do this, but 
> > > > > I have
> > > > > some concerns about the impact of this change.
> > > > > 
> > > > > Originally the split was done so that dib-run-parts and one of the
> > > > > os-*-config projects (looks like os-refresh-config) that depends on 
> > > > > it could
> > > > > be included in a stock distro cloud image without pulling in all of 
> > > > > dib.
> > > > > Note that it is still present in the requirements of orc: 
> > > > > https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> > > > > 
> > > > > Disk space in a distro cloud image is at a premium, so pulling in a 
> > > > > project
> > > > > like diskimage-builder to get one script out of it was not 
> > > > > acceptable, at
> > > > > least from what I was told at the time.
> > > > > 
> > > > > I believe this was done so a distro cloud image could be used with 
> > > > > Heat out
> > > > > of the box, hence the heat tag on this message.  I don't know exactly 
> > > > > what
> > > > > happened after we split out dib-utils, so I'm hoping someone can 
> > > > > confirm
> > > > > whether this requirement still exists.  I think Steve was the one who 
> > > > > made
> > > > > the original request.  There were a lot of Steves working on Heat at 
> > > > > the
> > > > > time though, so it's possible I'm wrong. ;-)
> > > > 
> > > > I don't think I'm the Steve you're referring to, but I do have some
> > > > additional info as a result of investigating this bug:
> > > > 
> > > > https://bugs.launchpad.net/tripleo/+bug/1673144
> > > > 
> > > > It appears we have three different versions of dib-run-parts on the
> > > > undercloud (and, presumably overcloud nodes) at the moment, which is a
> > > > pretty major headache from a maintenance/debugging perspective.
> > > > 
> > > 
> > > I looked at the bug and I think there may only be two different
> > > versions? The versions in /bin and /usr/bin seem to come from the same
> > > package (so I hope they are the same version). I don't understand what
> > > is going on with the ./lib version but that seems like either a local
> > > package / checkout or something else non-dib related.
> > > 
> > > Two versions is certainly less than ideal, though :).
> > 
> > No I think there are four versions, three unique:
> > 
> > (undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
> > dib-utils-0.0.11-1.el7.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
> > dib-utils-0.0.11-1.el7.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf
> > /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
> > diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
> > file /usr/local/bin/dib-run-parts is not owned by any package
> > 
> > /usr/bin/dib-run-parts and /bin/dib-run-parts are the same file, owned by
> > dib-utils
> > 
> > /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts is
> > owned by diskimage-builder
> > 
> > /usr/local/bin/dib-run-parts is the mystery file presumed from image
> > building
> > 
> > But the exciting thing from a rolling-out-bugfixes perspective is that
> > the
> > one actually running via o-r-c isn't either of the packaged versions
> > (doh!)
> > so we probably need to track down which element is installing it.
> > 
> > This is a little OT for this thread (sorry), but hopefully provides more
> > context around my concerns about creating another fork etc.
> > 
> > > > However we resolve this, *please* can we avoid permanently forking the
> > > > tool, as e.g in that bug, where do I send the patch to fix leaking
> > > > profiledir directories?  What package needs an update?  What is
> > > > installing
> > > > the script being run that's not owned by any package?
> > > > 
> > > > Yes, I know the answer to some of those questions, but I'm trying to
> > > > point
> > > > out duplicating this script and shipping it from multiple repos/packages
> > > > is
> > > > pretty horrible from a maintenance perspective, especially for new or
> > > > casual contributors.
> > > > 
> > > 
> > > I agree. You answered my previous question of whether os-refresh-config
> > > is still in use (sounds like it definitely is) so this complicates
> > > things a bit.
> > > 
> > > > If we have to fork it, I'd suggest we should rename the script to avoid
> > > > the
> > > > 

Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-16 Thread Clark Boylan
On Thu, Mar 16, 2017, at 09:46 AM, Steven Hardy wrote:
> On Thu, Mar 16, 2017 at 10:30:48AM -0500, Gregory Haynes wrote:
> > On Thu, Mar 16, 2017, at 05:18 AM, Steven Hardy wrote:
> > > On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> > > > While looking through the dib v2 changes after the feature branch was 
> > > > merged
> > > > to master, I noticed this commit[1], which bring dib-run-parts back 
> > > > into dib
> > > > itself.  Unfortunately I missed the original proposal to do this, but I 
> > > > have
> > > > some concerns about the impact of this change.
> > > > 
> > > > Originally the split was done so that dib-run-parts and one of the
> > > > os-*-config projects (looks like os-refresh-config) that depends on it 
> > > > could
> > > > be included in a stock distro cloud image without pulling in all of dib.
> > > > Note that it is still present in the requirements of orc: 
> > > > https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> > > > 
> > > > Disk space in a distro cloud image is at a premium, so pulling in a 
> > > > project
> > > > like diskimage-builder to get one script out of it was not acceptable, 
> > > > at
> > > > least from what I was told at the time.
> > > > 
> > > > I believe this was done so a distro cloud image could be used with Heat 
> > > > out
> > > > of the box, hence the heat tag on this message.  I don't know exactly 
> > > > what
> > > > happened after we split out dib-utils, so I'm hoping someone can confirm
> > > > whether this requirement still exists.  I think Steve was the one who 
> > > > made
> > > > the original request.  There were a lot of Steves working on Heat at the
> > > > time though, so it's possible I'm wrong. ;-)
> > > 
> > > I don't think I'm the Steve you're referring to, but I do have some
> > > additional info as a result of investigating this bug:
> > > 
> > > https://bugs.launchpad.net/tripleo/+bug/1673144
> > > 
> > > It appears we have three different versions of dib-run-parts on the
> > > undercloud (and, presumably overcloud nodes) at the moment, which is a
> > > pretty major headache from a maintenance/debugging perspective.
> > > 
> > 
> > I looked at the bug and I think there may only be two different
> > versions? The versions in /bin and /usr/bin seem to come from the same
> > package (so I hope they are the same version). I don't understand what
> > is going on with the ./lib version but that seems like either a local
> > package / checkout or something else non-dib related.
> > 
> > Two versions is certainly less than ideal, though :).
> 
> No I think there are four versions, three unique:
> 
> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch
> (undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch
> (undercloud) [stack@undercloud ~]$ rpm -qf
> /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
> diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch
> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
> file /usr/local/bin/dib-run-parts is not owned by any package
> 
> /usr/bin/dib-run-parts and /bin/dib-run-parts are the same file, owned by
> dib-utils
> 
> /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts is
> owned by diskimage-builder
> 
> /usr/local/bin/dib-run-parts is the mystery file presumed from image
> building
> 
> But the exciting thing from a rolling-out-bugfixes perspective is that
> the
> one actually running via o-r-c isn't either of the packaged versions
> (doh!)
> so we probably need to track down which element is installing it.
> 
> This is a little OT for this thread (sorry), but hopefully provides more
> context around my concerns about creating another fork etc.
> 
> > > However we resolve this, *please* can we avoid permanently forking the
> > > tool, as e.g in that bug, where do I send the patch to fix leaking
> > > profiledir directories?  What package needs an update?  What is
> > > installing
> > > the script being run that's not owned by any package?
> > > 
> > > Yes, I know the answer to some of those questions, but I'm trying to
> > > point
> > > out duplicating this script and shipping it from multiple repos/packages
> > > is
> > > pretty horrible from a maintenance perspective, especially for new or
> > > casual contributors.
> > > 
> > 
> > I agree. You answered my previous question of whether os-refresh-config
> > is still in use (sounds like it definitely is) so this complicates
> > things a bit.
> > 
> > > If we have to fork it, I'd suggest we should rename the script to avoid
> > > the
> > > confusion I outline in the bug above, e.g one script -> one repo -> one
> > > package?
> > 
> > I really like this idea of renaming the script in dib which should
> > clarify the source of each script and prevent conflicts, but this still
> > leaves the fork-related issues. If we go the route of just keeping the
> 

Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-16 Thread Steven Hardy
On Thu, Mar 16, 2017 at 10:30:48AM -0500, Gregory Haynes wrote:
> On Thu, Mar 16, 2017, at 05:18 AM, Steven Hardy wrote:
> > On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> > > While looking through the dib v2 changes after the feature branch was 
> > > merged
> > > to master, I noticed this commit[1], which bring dib-run-parts back into 
> > > dib
> > > itself.  Unfortunately I missed the original proposal to do this, but I 
> > > have
> > > some concerns about the impact of this change.
> > > 
> > > Originally the split was done so that dib-run-parts and one of the
> > > os-*-config projects (looks like os-refresh-config) that depends on it 
> > > could
> > > be included in a stock distro cloud image without pulling in all of dib.
> > > Note that it is still present in the requirements of orc: 
> > > https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> > > 
> > > Disk space in a distro cloud image is at a premium, so pulling in a 
> > > project
> > > like diskimage-builder to get one script out of it was not acceptable, at
> > > least from what I was told at the time.
> > > 
> > > I believe this was done so a distro cloud image could be used with Heat 
> > > out
> > > of the box, hence the heat tag on this message.  I don't know exactly what
> > > happened after we split out dib-utils, so I'm hoping someone can confirm
> > > whether this requirement still exists.  I think Steve was the one who made
> > > the original request.  There were a lot of Steves working on Heat at the
> > > time though, so it's possible I'm wrong. ;-)
> > 
> > I don't think I'm the Steve you're referring to, but I do have some
> > additional info as a result of investigating this bug:
> > 
> > https://bugs.launchpad.net/tripleo/+bug/1673144
> > 
> > It appears we have three different versions of dib-run-parts on the
> > undercloud (and, presumably overcloud nodes) at the moment, which is a
> > pretty major headache from a maintenance/debugging perspective.
> > 
> 
> I looked at the bug and I think there may only be two different
> versions? The versions in /bin and /usr/bin seem to come from the same
> package (so I hope they are the same version). I don't understand what
> is going on with the ./lib version but that seems like either a local
> package / checkout or something else non-dib related.
> 
> Two versions is certainly less than ideal, though :).

No I think there are four versions, three unique:

(undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
dib-utils-0.0.11-1.el7.noarch
(undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
dib-utils-0.0.11-1.el7.noarch
(undercloud) [stack@undercloud ~]$ rpm -qf 
/usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch
(undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
file /usr/local/bin/dib-run-parts is not owned by any package

/usr/bin/dib-run-parts and /bin/dib-run-parts are the same file, owned by
dib-utils

/usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts is
owned by diskimage-builder

/usr/local/bin/dib-run-parts is the mystery file presumed from image
building

But the exciting thing from a rolling-out-bugfixes perspective is that the
one actually running via o-r-c isn't either of the packaged versions (doh!)
so we probably need to track down which element is installing it.

This is a little OT for this thread (sorry), but hopefully provides more
context around my concerns about creating another fork etc.

> > However we resolve this, *please* can we avoid permanently forking the
> > tool, as e.g in that bug, where do I send the patch to fix leaking
> > profiledir directories?  What package needs an update?  What is
> > installing
> > the script being run that's not owned by any package?
> > 
> > Yes, I know the answer to some of those questions, but I'm trying to
> > point
> > out duplicating this script and shipping it from multiple repos/packages
> > is
> > pretty horrible from a maintenance perspective, especially for new or
> > casual contributors.
> > 
> 
> I agree. You answered my previous question of whether os-refresh-config
> is still in use (sounds like it definitely is) so this complicates
> things a bit.
> 
> > If we have to fork it, I'd suggest we should rename the script to avoid
> > the
> > confusion I outline in the bug above, e.g one script -> one repo -> one
> > package?
> 
> I really like this idea of renaming the script in dib which should
> clarify the source of each script and prevent conflicts, but this still
> leaves the fork-related issues. If we go the route of just keeping the
> current state (of there being a fork) I think we should do the rename.
> 
> The issue I spoke of (complications with depending on dib-utils when
> installing dib in a venv) I think came from a combination of this
> dependency and not requiring a package install (you used to be able to
> 

Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-16 Thread Gregory Haynes
On Thu, Mar 16, 2017, at 05:18 AM, Steven Hardy wrote:
> On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> > While looking through the dib v2 changes after the feature branch was merged
> > to master, I noticed this commit[1], which bring dib-run-parts back into dib
> > itself.  Unfortunately I missed the original proposal to do this, but I have
> > some concerns about the impact of this change.
> > 
> > Originally the split was done so that dib-run-parts and one of the
> > os-*-config projects (looks like os-refresh-config) that depends on it could
> > be included in a stock distro cloud image without pulling in all of dib.
> > Note that it is still present in the requirements of orc: 
> > https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> > 
> > Disk space in a distro cloud image is at a premium, so pulling in a project
> > like diskimage-builder to get one script out of it was not acceptable, at
> > least from what I was told at the time.
> > 
> > I believe this was done so a distro cloud image could be used with Heat out
> > of the box, hence the heat tag on this message.  I don't know exactly what
> > happened after we split out dib-utils, so I'm hoping someone can confirm
> > whether this requirement still exists.  I think Steve was the one who made
> > the original request.  There were a lot of Steves working on Heat at the
> > time though, so it's possible I'm wrong. ;-)
> 
> I don't think I'm the Steve you're referring to, but I do have some
> additional info as a result of investigating this bug:
> 
> https://bugs.launchpad.net/tripleo/+bug/1673144
> 
> It appears we have three different versions of dib-run-parts on the
> undercloud (and, presumably overcloud nodes) at the moment, which is a
> pretty major headache from a maintenance/debugging perspective.
> 

I looked at the bug and I think there may only be two different
versions? The versions in /bin and /usr/bin seem to come from the same
package (so I hope they are the same version). I don't understand what
is going on with the ./lib version but that seems like either a local
package / checkout or something else non-dib related.

Two versions is certainly less than ideal, though :).

> However we resolve this, *please* can we avoid permanently forking the
> tool, as e.g in that bug, where do I send the patch to fix leaking
> profiledir directories?  What package needs an update?  What is
> installing
> the script being run that's not owned by any package?
> 
> Yes, I know the answer to some of those questions, but I'm trying to
> point
> out duplicating this script and shipping it from multiple repos/packages
> is
> pretty horrible from a maintenance perspective, especially for new or
> casual contributors.
> 

I agree. You answered my previous question of whether os-refresh-config
is still in use (sounds like it definitely is) so this complicates
things a bit.

> If we have to fork it, I'd suggest we should rename the script to avoid
> the
> confusion I outline in the bug above, e.g one script -> one repo -> one
> package?

I really like this idea of renaming the script in dib which should
clarify the source of each script and prevent conflicts, but this still
leaves the fork-related issues. If we go the route of just keeping the
current state (of there being a fork) I think we should do the rename.

The issue I spoke of (complications with depending on dib-utils when
installing dib in a venv) I think came from a combination of this
dependency and not requiring a package install (you used to be able to
./bin/disk-image-create without installation). Now that we require
installation this may be less of an issue.

So the two reasonable options seem to be: 
* Deal with the forking cost. Not the biggest cost when you notice
dib-utils hasn't had a commit in over 3 months and that one was a robot
commit to add some github flair.
* Switch back to dib-utils in the other repo. I'm starting to prefer
this slightly given that it seems there's a valid use case for it to
live externally and our installation story has become a lot more clean.
AFAIK this shouldn't prevent us from making the script more portable,
but please correct me if there's something I'm missing.

> 
> Thanks!
> 
> Steve
> 

Cheers,
- Greg


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-16 Thread Steven Hardy
On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> While looking through the dib v2 changes after the feature branch was merged
> to master, I noticed this commit[1], which bring dib-run-parts back into dib
> itself.  Unfortunately I missed the original proposal to do this, but I have
> some concerns about the impact of this change.
> 
> Originally the split was done so that dib-run-parts and one of the
> os-*-config projects (looks like os-refresh-config) that depends on it could
> be included in a stock distro cloud image without pulling in all of dib.
> Note that it is still present in the requirements of orc: 
> https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> 
> Disk space in a distro cloud image is at a premium, so pulling in a project
> like diskimage-builder to get one script out of it was not acceptable, at
> least from what I was told at the time.
> 
> I believe this was done so a distro cloud image could be used with Heat out
> of the box, hence the heat tag on this message.  I don't know exactly what
> happened after we split out dib-utils, so I'm hoping someone can confirm
> whether this requirement still exists.  I think Steve was the one who made
> the original request.  There were a lot of Steves working on Heat at the
> time though, so it's possible I'm wrong. ;-)

I don't think I'm the Steve you're referring to, but I do have some
additional info as a result of investigating this bug:

https://bugs.launchpad.net/tripleo/+bug/1673144

It appears we have three different versions of dib-run-parts on the
undercloud (and, presumably overcloud nodes) at the moment, which is a
pretty major headache from a maintenance/debugging perspective.

However we resolve this, *please* can we avoid permanently forking the
tool, as e.g in that bug, where do I send the patch to fix leaking
profiledir directories?  What package needs an update?  What is installing
the script being run that's not owned by any package?

Yes, I know the answer to some of those questions, but I'm trying to point
out duplicating this script and shipping it from multiple repos/packages is
pretty horrible from a maintenance perspective, especially for new or
casual contributors.

If we have to fork it, I'd suggest we should rename the script to avoid the
confusion I outline in the bug above, e.g one script -> one repo -> one
package?

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-15 Thread Ian Wienand

On 03/16/2017 08:22 AM, Ben Nemec wrote:

Anyway, I don't know that anything is broken at the moment since I
believe dib-run-parts was brought over unchanged, but the retirement of
dib-utils was proposed in https://review.openstack.org/#/c/445617 and I
would like to resolve this question before we do anything like that.


The underlying motivation behind this was to isolate dib so we could
do things like re-implement dib-run-parts in posixy shell (for busybox
environments) or python, etc.

So my idea was we'd just leave dib-utils alone.  But it raises a good
point that both dib-utils and diskimage-builder are providing
dib-run-parts.  I think this is probably the main oversight here.

I've proposed [1] that makes dib use dib-run-parts from its private
library dir (rather than any globally installed version) and stops it
exporting the script to avoid conflict with dib-utils.  I think this
should allow everything to live in harmony?

-i

[1] https://review.openstack.org/#/c/446285/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-15 Thread Gregory Haynes
On Wed, Mar 15, 2017, at 04:22 PM, Ben Nemec wrote:
> While looking through the dib v2 changes after the feature branch was 
> merged to master, I noticed this commit[1], which bring dib-run-parts 
> back into dib itself.  Unfortunately I missed the original proposal to 
> do this, but I have some concerns about the impact of this change.
> 
> Originally the split was done so that dib-run-parts and one of the 
> os-*-config projects (looks like os-refresh-config) that depends on it 
> could be included in a stock distro cloud image without pulling in all 
> of dib.  Note that it is still present in the requirements of orc: 
> https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> 

I had forgotten about this, but you're completely correct - the
os-refresh-config phases are run via dib-run-parts. The reason for
moving dib-run-parts back in to dib was to simplify some of the
installation insanity we had going on, I want to say it was one reason
you couldn't run disk-image-create from a virtualenv without sourcing it
first.

> Disk space in a distro cloud image is at a premium, so pulling in a 
> project like diskimage-builder to get one script out of it was not 
> acceptable, at least from what I was told at the time.
> 
> I believe this was done so a distro cloud image could be used with Heat 
> out of the box, hence the heat tag on this message.  I don't know 
> exactly what happened after we split out dib-utils, so I'm hoping 
> someone can confirm whether this requirement still exists.  I think 
> Steve was the one who made the original request.  There were a lot of 
> Steves working on Heat at the time though, so it's possible I'm wrong.
> ;-)
> 
> Anyway, I don't know that anything is broken at the moment since I 
> believe dib-run-parts was brought over unchanged, but the retirement of 
> dib-utils was proposed in https://review.openstack.org/#/c/445617 and I 
> would like to resolve this question before we do anything like that.
> 

I think you're right in that nothing should be broken ATM since the API
is consistent. I agree that it doesn't make a lot of sense to retire
something which is depended on by other non-retired projects. The
biggest issue I can see with us leaving dib-utils in its current state
is there's the opportunity for the two implementations to drift and have
slightly different dib-run-parts APIs. Maybe we could prevent this by
deprecating dib-utils (or leaving a big warning of this tool is frozen
in the README) and leaving os-refresh-config as is. Although it isn't
ideal for os-refresh-config to depend on a deprecated tool I am not sure
anyone is making use of os-refresh-config currently so I am hesitant to
suggest we add back the complexity to DIB.

> Thanks.
> 
> -Ben
> 
> 1: 
> https://github.com/openstack/diskimage-builder/commit/d65678678ec0416550d768f323ceace4d0861bca
> 

Thanks!
- Greg

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-15 Thread Ben Nemec
While looking through the dib v2 changes after the feature branch was 
merged to master, I noticed this commit[1], which bring dib-run-parts 
back into dib itself.  Unfortunately I missed the original proposal to 
do this, but I have some concerns about the impact of this change.


Originally the split was done so that dib-run-parts and one of the 
os-*-config projects (looks like os-refresh-config) that depends on it 
could be included in a stock distro cloud image without pulling in all 
of dib.  Note that it is still present in the requirements of orc: 
https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5


Disk space in a distro cloud image is at a premium, so pulling in a 
project like diskimage-builder to get one script out of it was not 
acceptable, at least from what I was told at the time.


I believe this was done so a distro cloud image could be used with Heat 
out of the box, hence the heat tag on this message.  I don't know 
exactly what happened after we split out dib-utils, so I'm hoping 
someone can confirm whether this requirement still exists.  I think 
Steve was the one who made the original request.  There were a lot of 
Steves working on Heat at the time though, so it's possible I'm wrong. ;-)


Anyway, I don't know that anything is broken at the moment since I 
believe dib-run-parts was brought over unchanged, but the retirement of 
dib-utils was proposed in https://review.openstack.org/#/c/445617 and I 
would like to resolve this question before we do anything like that.


Thanks.

-Ben

1: 
https://github.com/openstack/diskimage-builder/commit/d65678678ec0416550d768f323ceace4d0861bca


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev