On Sun, Jan 08, 2017 at 02:45:28PM -0600, Gregory Haynes wrote: > On Fri, Jan 6, 2017, at 09:57 AM, Paul Belanger wrote: > > On Fri, Jan 06, 2017 at 09:48:31AM +0100, Andre Florath wrote: > > > Hello Paul, > > > > > > thank you very much for your contribution - it is very appreciated. > > > > > Seconded - I'm very excited for some effort to be put in to improving > the use case of making containers with DIB. Thanks :). > > > > You addressed a topic with your patch set that was IMHO not in a wide > > > focus: generating images for containers. The ideas in the patches are > > > good and should be implemented. > > > > > > Nevertheless I'm missing the concept behind your patches. What I saw > > > are a couple of (independent?) patches - and it looks that there is > > > one 'big goal' - but I did not really get it. My proposal is (as it > > > is done for other bigger changes or introducing new concepts) that > > > you write a spec for this first . That would help other people > > > (see e.g. Matthew) to use the same blueprint also for other > > > distributions. > > I strongly agree with the point that this is something were going to end > up repeating across many distros so we should make sure there's some > common patterns for doing so. A spec seems fine to me, but ideally the > end result involves some developer documentation. A spec is probably a > good place to get started on getting some consensus which we can turn in > to the dev docs. > This plan is to start with ubuntu, then move to debian, then fedora and finally centos. Fedora and CentOS are obviously harder, since a debootstrap tool doesn't exist.
> > Sure, I can write a spec if needed but the TL;DR is: > > > > Use diskimage-builder to build debootstrap --variant=minbase chroot, and > > nothing > > else. So I can then use take the generated tarball and do something else > > with > > it. > > > > > One possibility would be to classify different element sets and define > > > the dependency between them. E.g. to have a element class 'container' > > > which can be referenced by other classes, but is not able to reference > > > these (e.g. VM or hardware specific things). > > > > > It sounds like we need to step back a bit get a clear idea of how were > going to manage the full use case matrix of distro * (minimal / full) * > (container / vm / baremetal), which is something that would be nice to > get consensus on in a spec. This is something that keeps tripping up > both users and devs and I think adding containers to the matrix is sort > of a tipping point in terms of complexity so again, some docs after > figuring out our plan would be *awesome*. > > Currently we have distro-minimal elements which are minimal > vm/baremetal, and distro elements which actually are full vm/baremetal > elements. I assume by adding an element class you mean add a set of > distro-container elements? If so, I worry that we might be falling in to > a common dib antipattern of making distro-specific elements. I have a > alternate proposal: > > Lets make two elements: kernel, and minimal-userspace which, > respectively, install the kernel package and a minimal set of userspace > packages for dib to function (e.g. dependencies for dib-run-parts, > package-installs). The kernel package should be doable as basically a > package-installs and a pkg-map. The minimal-userspace element gets > tricky because it needs to install deps which are required for things > like package-installs to function (which is why the various distro > elements do this independently). Even so, I think it would be nice to > take care of installing these from within the chroot rather than from > outside (see https://review.openstack.org/#/c/392253/ for a good reason > why). If we do this then the minimal-userspace element can have some > common logic to enter the chroot as part of root.d and then install the > needed deps. > > The end result of this would be we have distro-minimal which depends on > kernel, minimal-userspace, and yum/debootstrap to build a vm/baremetal > capable image. We could also create a distro-container element which > only depends on minimal-userspace and yum/debootstrap and creates a > minimal container. The point being - the top level -container or > -minimal elements are basically convenience elements for exporting a few > vars and pulling in the proper elements at this point and the > elements/code are broken down by the functionality they provide rather > than use case. > To be honest, this is a ton of work, just to create an debootstrap 'operating system' element. I'm actually pretty happy how things look to day with our -minimal elements. But it will be an uphill battle to do the work you are asking. I can especially understand the need to refactor code and optimize, but just looking at the effort to create minimal / cloud elements, its been ongoing since Oct. 2015. We haven't even landed that.  https://review.openstack.org/#/c/211859/ > > > There are additional two major points: > > > > > > * IMHO you addressed only some elements that needs adaptions to be > > > able to used in containers. One element I stumbled over yesterday > > > is the base element: it is always included until you explicitly > > > exclude it. This base element depends on a complete init-system - > > > which is for a container unneeded overhead.  > > I think you're on the right track with removing base - we had consensus > a while back that it should go away but we never got around to it. The > big issue is going to be preserving backwards compat or making it part > of a major version bump and not too painful to upgrade to. I think we > can have this convo on the patchset, though. > > > > > Correct, for this I simply pass the -n flag to disk-image-create. This > > removes > > the need for include the base element. If we want to make a future > > optimization > > to remove or keep, I am okay with that. But the main goal for me is to > > include > > the new ubuntu-rootfs element with minimal disruption as possible. > > > > > > * Your patches add a lot complexity and code duplication. > > > This is not the way it should be (see , p 110, p 345). > > The main reason this was done, is yes there is some code duplication, but > > the > > because, this is done in the root.d phase. Moving this logic into > > another > > phase, then requires the need to install python into chroot, and then > > dpkg, > > dib-python, package-install, etc. This basically contaminants the > > pristine > > debootstrap environment, something I am trying hard not to do. I figure, > > 2 lines > > to delete stale data is fine. However, if there is an objection, we can > > remove > > it. Keep in mind, by deleting the cache we get the tarball size to 42Mb > > (down > > from 79Mb). > > I think my above proposal about a common userspace and kernel element > would solve most of the duplication issues. I am unclear about the not > wanting python and some other elements as part of the build. Currently > python as part of the target image is something we require for a large > portion of the dib functionality. It is possible to not have it as part > of the target image, but there's a significant cost to doing so. As > such, I'd like to know what the motivation is for this? Is it purely a > size reason, and if so, what sizes are we talking about for including > python? > A usecase would be a container needing to run go (not that I have one). Since containers are to be as minimal as possible, it is not a required dependency for the application. However, since for of the things I would build in a container does require python, I can then install it after the fact in the cleanup.d phase (simple-playbook). I'd like like the option to do it myself or not, this is why the ubuntu-rootfs element doesn't depend on package-install for example. > > > > > One reason is, that you do everything twice: once for Debian and > > > once for Ubuntu - and both in a (slightly) different way. > > Yes, sadly the debian elements came along after the ubuntu-minimal > > elements, > > with different people writing the code. For the most part, I've been > > trying to > > condense the code path between the 2, but we are slowly getting there. > > > > As you can see, the debian-rootfs element does now work correctly > > based on > > previous patches in the stack. > > > > However, I don't believe this is the stack to make things better between > > the 2 > > flavors. We can use the existing ubuntu-minimal and debian-minimal > > elements and > > iterate atop of that. One next steps is to address how we handle the > > sources.list file, between ubuntu and debian we do things differently. > > > >  https://review.openstack.org/#/c/414765/ > > > > > Please: factor out common code. > > > Please: improve code as you touch it. > > > > > > And three minor: > > > > > > * Release notes are missing (reno is your friend) > > > > > Sure, I can add release notes. > > > > > * Please do not introduce code which 'later on' can / should / will be > > > cleaned up. Do it correct right from the beginning.  > > > > > I can rebase code if needed. > > > > > * It looks that this is a bigger patch set - so maybe we should > > > include it in v2? > > > > > I'm not sure we need to wait for v2 (but I am biased). I've recently > > revamped > > our testing infra for diskimage-builder. We now build images, along with > > launching them with nodepool and SSHing into them. > > > > Side note, when is v2 landing? I know there has been issues with > > tripleo. > > > > We basically have the one patch (new block-device.d system) which needs > to land before we can cut an RC. As a result I would prefer to not add > things to v2 (so we can get on with the process of getting it released). > This patch is blocking on a final +A and then someone doing a merge of > master in to the v2 branch before the +A. I plan on doing this once I'm > back from holiday (1/11) if someone hasn't done so by then. Once this > merges the rough plan is to cut an RC, mail out to the list asking folks > for feedback, and cycle on that until we feel comfortable releasing. > > > > Kind regards > > > > > > Andre > > > > > > > > >  https://review.openstack.org/#/c/414728/ > > >  https://review.openstack.org/#/c/417310/ > > >  "Refactoring - Improving the Design of Existing Code", Martin > > > Fowler, Addison Wesley, Boston, 2011 > > >  > > > https://review.openstack.org/#/c/414728/8/elements/debootstrap-minimal/root.d/99-clean-up-cache > > >  https://review.openstack.org/#/c/413221/ > > > > > Thanks, > Greg > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev