Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
One more related idea related to real packages in TripleO. While I still think using packages is totally cool we may want to make an exception for systemd/upstart scripts. We have some non-standard ordering in our TripleO init scripts that is meaningful and blindly switching to a distro specific version would almost certainly cause issues. Dan - Original Message - > From: "James Slagle" > To: "OpenStack Development Mailing List (not for usage questions)" > > Sent: Wednesday, January 8, 2014 10:03:39 AM > Subject: Re: [openstack-dev] [TripleO] Installing from packages in > tripleo-image-elements > > On Tue, Jan 7, 2014 at 11:20 PM, Robert Collins > wrote: > > On 8 January 2014 12:18, James Slagle wrote: > >> Sure, the crux of the problem was likely that versions in the distro > >> were too old and they needed to be updated. But unless we take on > >> building the whole OS from source/git/whatever every time, we're > >> always going to have that issue. So, an additional benefit of > >> packages is that you can install a known good version of an OpenStack > >> component that is known to work with the versions of dependent > >> software you already have installed. > > > > The problem is that OpenStack is building against newer stuff than is > > in distros, so folk building on a packaging toolchain are going to > > often be in catchup mode - I think we need to anticipate package based > > environments running against releases rather than CD. > > I just don't see anyone not building on a packaging toolchain, given > that we're all running the distro of our choice and pip/virtualenv/etc > are installed from distro packages. Trying to isolate the building of > components with pip installed virtualenvs was still a problem. Short > of uninstalling the build tools packages from the cloud image and then > wget'ing the pip tarball, I don't think there would have been a good > way around this particular problem. Which, that approach may > certainly make some sense for a CD scenario. > > Agreed that packages against releases makes sense. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
We're in agreement. What little entry there might be in a system of such a small size would be entirely manageable by a single administrator... I care about that deployment, deeply, as that is how things like OpenStack take root in IT departments.. with somebody playing around. However, what I care more about is that when that deployment goes from POC to reality, it can scale up to tens of admins and thousands of machines. If it cannot, if the user finds themselves doing things manually and handling problems by poking packages out to small classes of machines, then we have failed and OpenStack will be very costly for any org to scale out. Excerpts from Fox, Kevin M's message of 2014-01-08 09:22:15 -0800: > Let me give you a more concrete example, since you still think one size fits > all here. > > I am using OpenStack on my home server now. In the past, I had one machine > with lots of services on it. At times, I would update one service and during > the update process, a different service would break. > > Last round of hardware purchasing got me an 8 core desktop processor with 16 > gigs of ram. Enough to give every service I have its own processor and 2 gigs > of ram. So, I decided to run OpenStack on the server to manage the service > vm's. > > The base server shares out my shared data with nfs, the vm's then re-export > it in various ways like samba, dlna to my ps3, etc. > > Now, I could create a golden image for each service type with everything all > setup and good to go. And infrastructure to constantly build updated ones. > > But in this case, grabbing Fedora cloud image or Ubuntu cloud image, and > starting up the service with heat and a couple of line cloud init telling it > to install just the package for the one service I need saves a ton of effort > and space. The complexity is totally on the distro folks and not me. Very > simple to maintain. > > I can get almost the stability of the golden image simply by pausing the > working service vm, spawning a new one, and only if its sane, switch to it > and delete the old. In fact, Heat is working towards (if not already done) > having Heat itself do this process for you. > > I'm all for golden images as a tool. We use them a lot. Like all tools > though, there isn't one "works for all cases best" tool. > > I hope this use case helps. > > Thanks, > Kevin ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Let me give you a more concrete example, since you still think one size fits all here. I am using OpenStack on my home server now. In the past, I had one machine with lots of services on it. At times, I would update one service and during the update process, a different service would break. Last round of hardware purchasing got me an 8 core desktop processor with 16 gigs of ram. Enough to give every service I have its own processor and 2 gigs of ram. So, I decided to run OpenStack on the server to manage the service vm's. The base server shares out my shared data with nfs, the vm's then re-export it in various ways like samba, dlna to my ps3, etc. Now, I could create a golden image for each service type with everything all setup and good to go. And infrastructure to constantly build updated ones. But in this case, grabbing Fedora cloud image or Ubuntu cloud image, and starting up the service with heat and a couple of line cloud init telling it to install just the package for the one service I need saves a ton of effort and space. The complexity is totally on the distro folks and not me. Very simple to maintain. I can get almost the stability of the golden image simply by pausing the working service vm, spawning a new one, and only if its sane, switch to it and delete the old. In fact, Heat is working towards (if not already done) having Heat itself do this process for you. I'm all for golden images as a tool. We use them a lot. Like all tools though, there isn't one "works for all cases best" tool. I hope this use case helps. Thanks, Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Wednesday, January 08, 2014 8:36 AM To: openstack-dev Subject: Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements Excerpts from Derek Higgins's message of 2014-01-08 02:11:09 -0800: > On 08/01/14 05:07, Clint Byrum wrote: > > Excerpts from Fox, Kevin M's message of 2014-01-07 16:27:35 -0800: > >> Another piece to the conversation I think is update philosophy. If > >> you are always going to require a new image and no customization after > >> build ever, ever, the messiness that source usually cause in the file > >> system image really doesn't matter. The package system allows you to > >> easily update, add, and remove packages bits at runtime cleanly. In > >> our experimenting with OpenStack, its becoming hard to determine > >> which philosophy is better. Golden Images for some things make a lot > >> of sense. For other random services, the maintenance of the Golden > >> Image seems to be too much to bother with and just installing a few > >> packages after image start is preferable. I think both approaches are > >> valuable. This may not directly relate to what is best for Triple-O > >> elements, but since we are talking philosophy anyway... > >> > > > > The golden image approach should be identical to the package approach if > > you are doing any kind of testing work-flow. > > > > "Just install a few packages" is how you end up with, as Robert said, > > "snowflakes". The approach we're taking with diskimage-builder should > > result in that image building extremely rapidly, even if you compiled > > those things from source. > > This is the part of your argument I don't understand, creating images > with packages is no more likely to result in snowflakes then creating > images from sources in git. > > You would build an image using packages and at the end of the build > process you can lock the package versions. Regardless of how the image > is built you can consider it a golden image. This image is then deployed > to your hosts and not changed. > > We would still be using diskimage-builder the main difference to the > whole process is we would end up with a image that has more packages > installed and no virtual envs. > I'm not saying building images from packages will encourage snowflakes. I'm saying installing and updating on systems using packages encourages snowflakes. Kevin was suggesting that the image workflow wouldn't fit for everything, and thus was opening up the "just install a few packages on a system" can of worms. I'm saying to Kevin, don't do that, just make your image work-flow tighter, and suggesting it is worth it to do that to avoid having snowflakes. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Jan Provaznik's message of 2014-01-08 03:00:19 -0800: > On 01/07/2014 09:01 PM, James Slagle wrote: > > Hi, > > > > I'd like to discuss some possible ways we could install the OpenStack > > components from packages in tripleo-image-elements. As most folks are > > probably aware, there is a "fork" of tripleo-image-elements called > > tripleo-puppet-elements which does install using packages, but it does > > so using Puppet to do the installation and for managing the > > configuration of the installed components. I'd like to kind of set > > that aside for a moment and just discuss how we might support > > installing from packages using tripleo-image-elements directly and not > > using Puppet. > > > > One idea would be to add support for a new type (or likely 2 new > > types: rpm and dpkg) to the source-repositories element. > > source-repositories already knows about the git, tar, and file types, > > so it seems somewhat natural to have additional types for rpm and > > dpkg. > > > > A complication with that approach is that the existing elements assume > > they're setting up everything from source. So, if we take a look at > > the nova element, and specifically install.d/74-nova, that script does > > stuff like install a nova service, adds a nova user, creates needed > > directories, etc. All of that wouldn't need to be done if we were > > installing from rpm or dpkg, b/c presumably the package would take > > care of all that. > > > > We could fix that by making the install.d scripts only run if you're > > installing a component from source. In that sense, it might make > > sense to add a new hook, source-install.d and only run those scripts > > if the type is a source type in the source-repositories configuration. > > We could then have a package-install.d to handle the installation > > from the packages type. The install.d hook could still exist to do > > things that might be common to the 2 methods. > > > > Thoughts on that approach or other ideas? > > > > I'm currently working on a patchset I can submit to help prove it out. > > But, I'd like to start discussion on the approach now to see if there > > are other ideas or major opposition to that approach. > > > > Hi James, > I think it would be really nice to be able install openstack+deps from > packages and many users (and cloud providers) would appreciate it. > > Among other things, with packages provided by a distro you get more > stability in compare to installing openstack from git repos and fetching > newest possible dependencies from pypi. > > In a real deployment setup I don't want to use "more-than-necessary" > newer packages/dependencies when building images - taking an example > from last days I wouldn't have to bother with newer pip package which > breaks image building. > Right, from this perspective, you want to run OpenStack stable releases. That should be fairly simple now by building images using the appropriate environment variables. However, we don't test that so it is likely to break as Icehouse diverges from Havana. So I think in addition to package-enabling, those who want to see TripleO work for stable releases should probably start looking at creating stable branches of t-i-e and t-h-t to build images and templates from starting at the icehouse time-frame. So given that I'd suggest that packages take a back seat to making TripleO part of the integrated release of OpenStack. Otherwise we'll just have stable releases for the distros who have packages that work with TripleO instead of for all distros. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Jay Dobies's message of 2014-01-08 08:09:51 -0800: > There were so many places in this thread that I wanted to jump in on as > I caught up, it makes sense to just summarize things in once place > instead of a half dozen quoted replies. > > I agree with the sentiments about flexibility. Regardless of my personal > preference on source v. packages, it's been my experience that the > general mindset of production deployment is that new ideas move slowly. > Admins are set in their ways and policies are in place on how things are > consumed. > > Maybe the newness of all things cloud-related and image-based management > for scale is a good time to shift the mentality out of packages (again, > I'm not suggesting whether or not it should be shifted). But I worry > about adoption if we don't provide an option for people to use blessed > distro packages, either because of company policy or years of habit and > bias. If done correctly, there's no difference between a package and a > particular tag in a source repository, but there is a psychological > component there that I think we need to account for, assuming someone is > willing to bite off the implementation costs (which is sounds like there > is). > Thanks for your thoughts Jay. I agree, what we're doing is kind of weird sounding. Not everybody will be on-board with their OpenStack cloud being wildly different from their existing systems. We definitely need to do work to make it easy for them to get on the new train of thinking one step at a time. Just having an OpenStack cloud will do a lot for any org that has none. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from James Slagle's message of 2014-01-08 07:03:39 -0800: > On Tue, Jan 7, 2014 at 11:20 PM, Robert Collins > wrote: > > On 8 January 2014 12:18, James Slagle wrote: > >> Sure, the crux of the problem was likely that versions in the distro > >> were too old and they needed to be updated. But unless we take on > >> building the whole OS from source/git/whatever every time, we're > >> always going to have that issue. So, an additional benefit of > >> packages is that you can install a known good version of an OpenStack > >> component that is known to work with the versions of dependent > >> software you already have installed. > > > > The problem is that OpenStack is building against newer stuff than is > > in distros, so folk building on a packaging toolchain are going to > > often be in catchup mode - I think we need to anticipate package based > > environments running against releases rather than CD. > > I just don't see anyone not building on a packaging toolchain, given > that we're all running the distro of our choice and pip/virtualenv/etc > are installed from distro packages. Trying to isolate the building of > components with pip installed virtualenvs was still a problem. Short > of uninstalling the build tools packages from the cloud image and then > wget'ing the pip tarball, I don't think there would have been a good > way around this particular problem. Which, that approach may > certainly make some sense for a CD scenario. > I will definitely concede that we find problems at a high rate during image builds, and that we would not if we just waited for others to solve those problems. However, when we do solve those problems, we solve them for everyone downstream from us. That is one reason it is so desirable to keep our work in TripleO as far upstream as possible. Package work is inherently downstream. Also it is worth noting that problems at image build time are much simpler to handle, because they happen on a single machine generally. That is one reason I down play those issues. For anyone not interested in running CD, we have the release process to handle such problems and they should _never_ see any of these issues, whether running from packages or on stable branches in the git repos. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Derek Higgins's message of 2014-01-08 02:11:09 -0800: > On 08/01/14 05:07, Clint Byrum wrote: > > Excerpts from Fox, Kevin M's message of 2014-01-07 16:27:35 -0800: > >> Another piece to the conversation I think is update philosophy. If > >> you are always going to require a new image and no customization after > >> build ever, ever, the messiness that source usually cause in the file > >> system image really doesn't matter. The package system allows you to > >> easily update, add, and remove packages bits at runtime cleanly. In > >> our experimenting with OpenStack, its becoming hard to determine > >> which philosophy is better. Golden Images for some things make a lot > >> of sense. For other random services, the maintenance of the Golden > >> Image seems to be too much to bother with and just installing a few > >> packages after image start is preferable. I think both approaches are > >> valuable. This may not directly relate to what is best for Triple-O > >> elements, but since we are talking philosophy anyway... > >> > > > > The golden image approach should be identical to the package approach if > > you are doing any kind of testing work-flow. > > > > "Just install a few packages" is how you end up with, as Robert said, > > "snowflakes". The approach we're taking with diskimage-builder should > > result in that image building extremely rapidly, even if you compiled > > those things from source. > > This is the part of your argument I don't understand, creating images > with packages is no more likely to result in snowflakes then creating > images from sources in git. > > You would build an image using packages and at the end of the build > process you can lock the package versions. Regardless of how the image > is built you can consider it a golden image. This image is then deployed > to your hosts and not changed. > > We would still be using diskimage-builder the main difference to the > whole process is we would end up with a image that has more packages > installed and no virtual envs. > I'm not saying building images from packages will encourage snowflakes. I'm saying installing and updating on systems using packages encourages snowflakes. Kevin was suggesting that the image workflow wouldn't fit for everything, and thus was opening up the "just install a few packages on a system" can of worms. I'm saying to Kevin, don't do that, just make your image work-flow tighter, and suggesting it is worth it to do that to avoid having snowflakes. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
There were so many places in this thread that I wanted to jump in on as I caught up, it makes sense to just summarize things in once place instead of a half dozen quoted replies. I agree with the sentiments about flexibility. Regardless of my personal preference on source v. packages, it's been my experience that the general mindset of production deployment is that new ideas move slowly. Admins are set in their ways and policies are in place on how things are consumed. Maybe the newness of all things cloud-related and image-based management for scale is a good time to shift the mentality out of packages (again, I'm not suggesting whether or not it should be shifted). But I worry about adoption if we don't provide an option for people to use blessed distro packages, either because of company policy or years of habit and bias. If done correctly, there's no difference between a package and a particular tag in a source repository, but there is a psychological component there that I think we need to account for, assuming someone is willing to bite off the implementation costs (which is sounds like there is). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 11:20 PM, Robert Collins wrote: > On 8 January 2014 12:18, James Slagle wrote: >> Sure, the crux of the problem was likely that versions in the distro >> were too old and they needed to be updated. But unless we take on >> building the whole OS from source/git/whatever every time, we're >> always going to have that issue. So, an additional benefit of >> packages is that you can install a known good version of an OpenStack >> component that is known to work with the versions of dependent >> software you already have installed. > > The problem is that OpenStack is building against newer stuff than is > in distros, so folk building on a packaging toolchain are going to > often be in catchup mode - I think we need to anticipate package based > environments running against releases rather than CD. I just don't see anyone not building on a packaging toolchain, given that we're all running the distro of our choice and pip/virtualenv/etc are installed from distro packages. Trying to isolate the building of components with pip installed virtualenvs was still a problem. Short of uninstalling the build tools packages from the cloud image and then wget'ing the pip tarball, I don't think there would have been a good way around this particular problem. Which, that approach may certainly make some sense for a CD scenario. Agreed that packages against releases makes sense. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On 01/07/2014 09:01 PM, James Slagle wrote: Hi, I'd like to discuss some possible ways we could install the OpenStack components from packages in tripleo-image-elements. As most folks are probably aware, there is a "fork" of tripleo-image-elements called tripleo-puppet-elements which does install using packages, but it does so using Puppet to do the installation and for managing the configuration of the installed components. I'd like to kind of set that aside for a moment and just discuss how we might support installing from packages using tripleo-image-elements directly and not using Puppet. One idea would be to add support for a new type (or likely 2 new types: rpm and dpkg) to the source-repositories element. source-repositories already knows about the git, tar, and file types, so it seems somewhat natural to have additional types for rpm and dpkg. A complication with that approach is that the existing elements assume they're setting up everything from source. So, if we take a look at the nova element, and specifically install.d/74-nova, that script does stuff like install a nova service, adds a nova user, creates needed directories, etc. All of that wouldn't need to be done if we were installing from rpm or dpkg, b/c presumably the package would take care of all that. We could fix that by making the install.d scripts only run if you're installing a component from source. In that sense, it might make sense to add a new hook, source-install.d and only run those scripts if the type is a source type in the source-repositories configuration. We could then have a package-install.d to handle the installation from the packages type. The install.d hook could still exist to do things that might be common to the 2 methods. Thoughts on that approach or other ideas? I'm currently working on a patchset I can submit to help prove it out. But, I'd like to start discussion on the approach now to see if there are other ideas or major opposition to that approach. Hi James, I think it would be really nice to be able install openstack+deps from packages and many users (and cloud providers) would appreciate it. Among other things, with packages provided by a distro you get more stability in compare to installing openstack from git repos and fetching newest possible dependencies from pypi. In a real deployment setup I don't want to use "more-than-necessary" newer packages/dependencies when building images - taking an example from last days I wouldn't have to bother with newer pip package which breaks image building. Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On 08/01/14 05:07, Clint Byrum wrote: > Excerpts from Fox, Kevin M's message of 2014-01-07 16:27:35 -0800: >> Another piece to the conversation I think is update philosophy. If >> you are always going to require a new image and no customization after >> build ever, ever, the messiness that source usually cause in the file >> system image really doesn't matter. The package system allows you to >> easily update, add, and remove packages bits at runtime cleanly. In >> our experimenting with OpenStack, its becoming hard to determine >> which philosophy is better. Golden Images for some things make a lot >> of sense. For other random services, the maintenance of the Golden >> Image seems to be too much to bother with and just installing a few >> packages after image start is preferable. I think both approaches are >> valuable. This may not directly relate to what is best for Triple-O >> elements, but since we are talking philosophy anyway... >> > > The golden image approach should be identical to the package approach if > you are doing any kind of testing work-flow. > > "Just install a few packages" is how you end up with, as Robert said, > "snowflakes". The approach we're taking with diskimage-builder should > result in that image building extremely rapidly, even if you compiled > those things from source. This is the part of your argument I don't understand, creating images with packages is no more likely to result in snowflakes then creating images from sources in git. You would build an image using packages and at the end of the build process you can lock the package versions. Regardless of how the image is built you can consider it a golden image. This image is then deployed to your hosts and not changed. We would still be using diskimage-builder the main difference to the whole process is we would end up with a image that has more packages installed and no virtual envs. > > What I'm suggesting is that you still need to test everything every > change you make, so you should just use the same work flow for > everything. > >> Again though, I think if you wish to make the argument that packages >> are undesirable, then ALL packages are probably undesirable for the same >> reasons. Right? Why not make elements for all dependencies, instead >> of using distro packages to get you 90% of the way there and then >> using source just for OpenStack bits. If you always want the newest, >> latest greatest Neutron, don't you want the newest VSwitch too? I'd >> argue though there is a point of diminishing returns with source that >> packages fill. Then the argument is where is the point. Some folks think >> the point is all the way over to "just use packages for everything". >> > > I've already stated that the distro is great for utilities, libraries, > drivers, kernels, etc. These are platform things, not application > things. OpenVSwitch _is_ part of the application, and I would entertain > building it for images if we found ourselves in need of hyper-advanced > features. > > What I've bee suggesting is that if I'm going to do the work to get > "latest OpenVSwitch", if I do it in a source->image way, I don't have to > repeat it for every distribution's packaging tool chain. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Jay Pipes's message of 2014-01-07 21:26:44 -0800: > On Wed, 2014-01-08 at 17:20 +1300, Robert Collins wrote: > > On 8 January 2014 12:18, James Slagle wrote: > > > Sure, the crux of the problem was likely that versions in the distro > > > were too old and they needed to be updated. But unless we take on > > > building the whole OS from source/git/whatever every time, we're > > > always going to have that issue. So, an additional benefit of > > > packages is that you can install a known good version of an OpenStack > > > component that is known to work with the versions of dependent > > > software you already have installed. > > > > The problem is that OpenStack is building against newer stuff than is > > in distros, so folk building on a packaging toolchain are going to > > often be in catchup mode - I think we need to anticipate package based > > environments running against releases rather than CD. > > It's about matching the expectations of the end user (deployer). If they > lean towards a CD model, then git based OpenStack deployments are > clearly a necessity -- otherwise we'd need to maintain a set of package > archives built (and tested!) for every project and every commit to > master. > > If they lean towards a more traditional release model, then OpenStack > packages maintained (and tested!) by the distros are a better fit for > the end user. > > However... > > Both camps should be able to experience the benefits of having an > OpenStack undercloud building an OpenStack overcloud, without forcing > the end user to adopt a methodology they may not yet be comfortable > with. > Jay's statements highlight that I have not been clear during this thread. I don't mean to force any of the methods on anyone, and agree with Jay that we should be able to leverage OpenStack to deploy OpenStack in many different ways. My reason for questioning the value of packages is that we have limited resources, and I don't want to direct them toward an effort solely because "thats what we've always done". I'm a packager myself, so there is this desire, sometimes, to just go back to the tools I'm comfortable with. But if there are developers who want and/or need to put time into using packages then by all means, I welcome you with open arms. I think the thread has proven to me that there is value in giving people the option to use TripleO's tools with packages. > As an aside, Fuel has an overcloud builder that uses Puppet to deploy > OpenStack with packages [1]. I understand the Fuel dev team at Mirantis > is keen to join forces with the Triple-O contributor community and > reduce duplicative efforts. This may be the perfect place to pull some > practices from Fuel to enable some flexibility in how > tripleo-image-elements constructs things. > > If you want a good indication of how much overlap there is, just look at > the list of puppet modules in Fuel [2] vs. the list of elements in t-i-e > [3]. > Nice comparison. > Sure, t-i-e is Bash scripts and Fuel is Puppet manifests, but they are > trying to do the same fundamental thing: produce an Overcloud entirely > through automation. There are definite advantages to both approaches. > Would be great to put our heads together and get the best of both > worlds. I think the end user would benefit. > Right, we are quite well aligned there. Where we run afoul of each other is that Puppet is a religion, and thus Fuel's puppet pieces are alienating the Chef believers. We'd rather not do that in TripleO. Reminding myself of this is why I realize above that "not packages" is also a religion, so I need to make sure I remain secular. I think we should actually use this as an example of where to plug Puppet in on top of the TripleO native tools which are emphatically not ever going to be a replacement for Puppet beyond the minimal needed to get a running, testable cloud. And then perhaps the Chef clergy can do the same, and we'll have a nice example of how to layer tools on top of TripleO's modular design. > > [1] Just one example... here is the Glance installation by package: > https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/glance/manifests/registry.pp#L76 > [2] > https://github.com/stackforge/fuel-library/tree/master/deployment/puppet > [3] > https://github.com/openstack/tripleo-image-elements/tree/master/elements > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Wed, 2014-01-08 at 17:20 +1300, Robert Collins wrote: > On 8 January 2014 12:18, James Slagle wrote: > > Sure, the crux of the problem was likely that versions in the distro > > were too old and they needed to be updated. But unless we take on > > building the whole OS from source/git/whatever every time, we're > > always going to have that issue. So, an additional benefit of > > packages is that you can install a known good version of an OpenStack > > component that is known to work with the versions of dependent > > software you already have installed. > > The problem is that OpenStack is building against newer stuff than is > in distros, so folk building on a packaging toolchain are going to > often be in catchup mode - I think we need to anticipate package based > environments running against releases rather than CD. It's about matching the expectations of the end user (deployer). If they lean towards a CD model, then git based OpenStack deployments are clearly a necessity -- otherwise we'd need to maintain a set of package archives built (and tested!) for every project and every commit to master. If they lean towards a more traditional release model, then OpenStack packages maintained (and tested!) by the distros are a better fit for the end user. However... Both camps should be able to experience the benefits of having an OpenStack undercloud building an OpenStack overcloud, without forcing the end user to adopt a methodology they may not yet be comfortable with. As an aside, Fuel has an overcloud builder that uses Puppet to deploy OpenStack with packages [1]. I understand the Fuel dev team at Mirantis is keen to join forces with the Triple-O contributor community and reduce duplicative efforts. This may be the perfect place to pull some practices from Fuel to enable some flexibility in how tripleo-image-elements constructs things. If you want a good indication of how much overlap there is, just look at the list of puppet modules in Fuel [2] vs. the list of elements in t-i-e [3]. Sure, t-i-e is Bash scripts and Fuel is Puppet manifests, but they are trying to do the same fundamental thing: produce an Overcloud entirely through automation. There are definite advantages to both approaches. Would be great to put our heads together and get the best of both worlds. I think the end user would benefit. All the best, -jay [1] Just one example... here is the Glance installation by package: https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/glance/manifests/registry.pp#L76 [2] https://github.com/stackforge/fuel-library/tree/master/deployment/puppet [3] https://github.com/openstack/tripleo-image-elements/tree/master/elements ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Fox, Kevin M's message of 2014-01-07 16:27:35 -0800: > Another piece to the conversation I think is update philosophy. If > you are always going to require a new image and no customization after > build ever, ever, the messiness that source usually cause in the file > system image really doesn't matter. The package system allows you to > easily update, add, and remove packages bits at runtime cleanly. In > our experimenting with OpenStack, its becoming hard to determine > which philosophy is better. Golden Images for some things make a lot > of sense. For other random services, the maintenance of the Golden > Image seems to be too much to bother with and just installing a few > packages after image start is preferable. I think both approaches are > valuable. This may not directly relate to what is best for Triple-O > elements, but since we are talking philosophy anyway... > The golden image approach should be identical to the package approach if you are doing any kind of testing work-flow. "Just install a few packages" is how you end up with, as Robert said, "snowflakes". The approach we're taking with diskimage-builder should result in that image building extremely rapidly, even if you compiled those things from source. What I'm suggesting is that you still need to test everything every change you make, so you should just use the same work flow for everything. > Again though, I think if you wish to make the argument that packages > are undesirable, then ALL packages are probably undesirable for the same > reasons. Right? Why not make elements for all dependencies, instead > of using distro packages to get you 90% of the way there and then > using source just for OpenStack bits. If you always want the newest, > latest greatest Neutron, don't you want the newest VSwitch too? I'd > argue though there is a point of diminishing returns with source that > packages fill. Then the argument is where is the point. Some folks think > the point is all the way over to "just use packages for everything". > I've already stated that the distro is great for utilities, libraries, drivers, kernels, etc. These are platform things, not application things. OpenVSwitch _is_ part of the application, and I would entertain building it for images if we found ourselves in need of hyper-advanced features. What I've bee suggesting is that if I'm going to do the work to get "latest OpenVSwitch", if I do it in a source->image way, I don't have to repeat it for every distribution's packaging tool chain. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On 8 January 2014 12:18, James Slagle wrote: > I'm reminded of when I first started looking at TripleO there were a > few issues with installing from git (I'll say that from now on :) > related to all the python distribute -> setuptools migration. Things > like if you're base cloud image had the wrong version of pip you > couldn't migrate to setuptools cleanly. Then you had to run the > setuptools update twice, once to get the distribute legacy wrapper and > then again to latest setuptools. If I recall there were other > problems with virtualenv incompatibilities as well. > > Arguably, installing from packages would have made that easier and less > complex. We should have that argument with a beverage and plenty of time ;). Certainly it was an automated fail - but automation detected the issues, and *if* we were in the gate, the changes that were done in OpenStack to trigger [most] of those issues would not have landed at all. > Sure, the crux of the problem was likely that versions in the distro > were too old and they needed to be updated. But unless we take on > building the whole OS from source/git/whatever every time, we're > always going to have that issue. So, an additional benefit of > packages is that you can install a known good version of an OpenStack > component that is known to work with the versions of dependent > software you already have installed. The problem is that OpenStack is building against newer stuff than is in distros, so folk building on a packaging toolchain are going to often be in catchup mode - I think we need to anticipate package based environments running against releases rather than CD. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On 8 January 2014 09:23, Clint Byrum wrote: > What would be the benefit of using packages? - Defense in depth - tests at the package build time + tests of the image. - Enumeration of installed software using the admins expected tooling (dpkg / rpm etc) - Familiarity > We've specifically avoided packages because they complect[1] configuration > and system state management with software delivery. The recent friction > we've seen with MySQL is an example where the packages are not actually > helping us, they're hurting us because they encode too much configuration > instead of just delivering binaries. Thats not why I'd say we're avoiding packages upstream :) I would say that running [professional quality] repositories is non-trivial, not directly related to our goals (because we have code -> image -> test -> deploy) and thus not something we've had incentive to do. Add in the numerous distros we support now and it becomes a significant burden for little benefit to our direct goals, and little benefit to our users. But if someone *has* packages, I see no harm (from the code->image->test->deploy) cycle to them being used, and it certainly helps vendors who are invested in packages to share effort between teams installing OpenStack in more traditional ways and those getting on the TripleO pipeline. I'm not disputing the possible downsides intrinsic to packages, particularly the not-at-scale temptation to create non automated snowflakes :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On 8 January 2014 12:26, Fox, Kevin M wrote: > One of the major features using a distro over upstream gives is integration. > rhel6 behaves differently then ubuntu 13.10. Sometimes it takes a while to > fix upstream for a given distro, and even then it may not even be accepted > because "the distro's too old, go away". Packages allow a distro to make sure > all the pieces can work together properly. And quickly patch things when > needed. Its kind of a fork but not quite the same thing. The distro > integrates not just OpenStack, but all its dependencies and their > dependencies all the way up. For example there can be subtle issues if > neutron, open vswitch and the kernel are out of sync. It is the integration > that folks like about distro's. "I can trust that since it came from X, all > the pieces should work together, or I know who to call". > > The only way I can think of to get the same stability out of just source is > for Triple-O to provide a whole source distro and test it, itself. More work > then it probably wants to do. Or pick just one distro and support source only > on that. Though if you pick the wrong distro, then you get into trust issues > and religious wars. :/ It is precisely that integration tested confidence that inspired the TripleO design: we don't know if a given combination of components work unless we've tested it. So TripleO is about the lifecycle: code -> image -> test -> deploy. What is deployed is what was tested. Where we source what was tested - packages or pypi or git - is irrelevant to the test results. I think we need to support everything that someone wants to offer up patches to make work (and ongoing support for whatever approach that is) because there are many use cases for users: if they get OpenStack using TripleO from e.g. Mirantis, or RedHat, or upstream, they will want to be using the install mechanism preferred by their service provider - and for TripleO to be widely applicable, we need to permit service providers to dial their own story. We're at the top of a waterfall of pull-and-build-and-test-and-distribute - we don't get to pick Chef vs Puppet vs nothing, or packages vs git vs pypi, or RHEL vs Ubuntu or Xen vs Kvm or Ceph vs GlusterFS. We do need to start with one thing and then add more backends - balance generalising everything with the cost ofa dding plug points after things are in use. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from James Slagle's message of 2014-01-07 15:18:00 -0800: > On Tue, Jan 7, 2014 at 6:04 PM, Clint Byrum wrote: > > Excerpts from Chris Jones's message of 2014-01-07 14:43:31 -0800: > >> Hi > >> > >> > On 7 Jan 2014, at 22:18, Clint Byrum wrote: > >> > Packages do the opposite, > >> > and encourage entropy by promising to try and update software > >> > >> Building with packages doesn't require updating running systems with > >> packages and more than building with git requires updating running systems > >> with git pull. > >> One can simply build (and test!) a new image with updated packages and > >> rebuild/takeover nodes. > >> > > > > Indeed, however one can _more_ simply build an image without package > > tooling... and they will be more similar across multiple platforms. > > > > My question still stands, what are the real advantages? So far the only > > one that matters to me is "makes it easier for people to think about > > using it." > > I'm reminded of when I first started looking at TripleO there were a > few issues with installing from git (I'll say that from now on :) > related to all the python distribute -> setuptools migration. Things > like if you're base cloud image had the wrong version of pip you > couldn't migrate to setuptools cleanly. Then you had to run the > setuptools update twice, once to get the distribute legacy wrapper and > then again to latest setuptools. If I recall there were other > problems with virtualenv incompatibilities as well. > > Arguably, installing from packages would have made that easier and less > complex. No argument, it would have been easier. But it would not have been less complex. The complexity would have been obscured. It really would have just deferred the problem to the distro. That may be a good thing, as the distro knows how to solve its own problems. But then Fedora solves it, and Ubuntu solves it, and Debian solves it... Or, OpenStack solves it, once, and OpenStack's users roll on no matter what they choose for distro. > > Sure, the crux of the problem was likely that versions in the distro > were too old and they needed to be updated. But unless we take on > building the whole OS from source/git/whatever every time, we're > always going to have that issue. So, an additional benefit of > packages is that you can install a known good version of an OpenStack > component that is known to work with the versions of dependent > software you already have installed. > I disagree on the "all or nothing" argument. We can have a pre-packaged OS that implements stable interfaces like POSIX, python-2.7 and even "git", and we can have a fast moving application like OpenStack riding on top of that in a self contained application container like a virtualenv or even a Docker container. The Linux distro model is _really_ good at providing an OS, tools and libraries. I am not so convinced that the distro model is actually any good for providing applications. I say that as one of the current maintainers of MySQL, which mixes libraries and applications, in Debian. The pain that the MySQL packaging team goes through just to have the thing work similar to postgresql and apache is a little bit crazy-inducing, so please excuse me when I shake up the bottle and spray crazy all over the list. :) Also, "known good" is incredibly subjective, as "good" implies a set of expectations that are being met. But whatever that "good" means is fairly poorly defined and probably not automated. :P ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 6:45 PM, Clint Byrum wrote: > Excerpts from James Slagle's message of 2014-01-07 15:03:33 -0800: >> On Tue, Jan 7, 2014 at 5:18 PM, Clint Byrum wrote: >> > Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800: >> >> On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: > Image proliferation is far easier to measure than package proliferation. > But, that is not really the point. If you're managing it right, you don't have package proliferation, any more than you do image proliferation. But, if that's no longer the point, then we can move on :). > The point is that we have a tool for software distribution that fits into > our system management approach, and the distro packages do not take that > system management approach into account. > > So if you can't use the distro packages as-is, I'm questioning what the > actual benefit of using them at all is. Well, I can't argue that if we install from packages, and then have to hack up the installed files to make them work with our approach, that is not ideal and not really sane. Personally, it would be great if the distro packages had support for the image based approach. And truly, if OpenStack is adopting the image based installation and deployment mechanism as "the way", then the Installation guide isn't even going to say install from packages anymore on top of your base OS. It's just going to be all TripleO. And, I think the distros would have to adapt to that. I don't honestly know how much you could use the current distro packages as-is. But, I'm sure I'll find out soon enough. > >> > >> >> > We've specifically avoided packages because they complect[1] >> >> > configuration >> >> > and system state management with software delivery. The recent friction >> >> > we've seen with MySQL is an example where the packages are not actually >> >> > helping us, they're hurting us because they encode too much >> >> > configuration >> >> > instead of just delivering binaries. >> >> >> >> We're trying to do something fairly specific with the read only / >> >> partition. You're right, most packages aren't going to handle that >> >> well. So, yes you have a point from that perspective. >> >> >> > >> > Readonly / is a really important feature of the deployment we're aiming >> > at. Doing it with packages is quite possible. My point in asking why >> > bother with packages is that when you have an entire image that has been >> > verified and is known to work, what advantage does having a package for >> > everything actually bring. >> >> Because distro packages are known to work, and thus you get higher >> confidence from any image constructed from said packages. At least, I >> would, as opposed to installing from source (or "from git" as you say >> below :). It's the same reason I want to use a packaged kernel >> instead of compiling it from source. The benefit of the package is >> not just in the compiling. It's in the known good version and >> compatibility with other known good versions I want to use. >> > > I would disagree with "known to work". They are known to have been > tested at some level. But IMO "known to work" requires testing _with > your workload_. So this is just semantics around "known to work". Definitely you have to test in your own environment before deploying anything. I more meant "known to be compatible", "known to be supported", "known to work on certain hardware". Or "advertised to". Things of that nature. But, none of those imply you don't test. Also, the inverse can be equally valuable. These versions do *not* work on this hardware, etc. > Since you have to test your workload, why bother with the distro packages > when you can get the upstream software and testing suite directly. > >> Am I going to implicitly trust any packages blindly or completely? Of >> course not. But, there is some confidence there in that the distro has >> done some testing and said these versions are compatible, etc. >> > > I think that confidence is misplaced and unnecessary. Certainly if you set the expectation that that nothing is "known to work", then no one is going to have any confidence that it does. OpenStack does releases of projects that are in some form of a known good state. I would think most people would reasonably expect that downloading that release would likely work better vs. grabbing from git 2 weeks into a new development cycle. Meaning, I have some confidence that OpenStack as a community has done some testing. We have qa, gates, unit and functional tests, etc. I don't think that confidence is misplaced or unnecessary. If a distro says a set of OpenStack packages work with a given version of their OS, then that confidence is not misplaced either IMO. > We provide test > suites to users and we will encourage users to test their own things. I > imagine some will also ship packaged products based on TripleO that will > also be tested as a whole, not as individual packages. This is a new
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Another piece to the conversation I think is update philosophy. If you are always going to require a new image and no customization after build ever, ever, the messiness that source usually cause in the file system image really doesn't matter. The package system allows you to easily update, add, and remove packages bits at runtime cleanly. In our experimenting with OpenStack, its becoming hard to determine which philosophy is better. Golden Images for some things make a lot of sense. For other random services, the maintenance of the Golden Image seems to be too much to bother with and just installing a few packages after image start is preferable. I think both approaches are valuable. This may not directly relate to what is best for Triple-O elements, but since we are talking philosophy anyway... Again though, I think if you wish to make the argument that packages are undesirable, then ALL packages are probably undesirable for the same reasons. Right? Why not make elements for all dependencies, instead of using distro packages to get you 90% of the way there and then using source just for OpenStack bits. If you always want the newest, latest greatest Neutron, don't you want the newest VSwitch too? I'd argue though there is a point of diminishing returns with source that packages fill. Then the argument is where is the point. Some folks think the point is all the way over to "just use packages for everything". I still think using distro packages buys you a lot, even if you are just using them to create a static golden image. Thanks, Kevin From: James Slagle [james.sla...@gmail.com] Sent: Tuesday, January 07, 2014 3:59 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements On Tue, Jan 7, 2014 at 6:12 PM, Clint Byrum wrote: > Excerpts from Fox, Kevin M's message of 2014-01-07 13:11:13 -0800: >> I was going to stay silent on this one, but since you asked... >> >> /me puts his customer hat on >> >> We source OpenStack from RDO for the packages and additional integration >> testing that comes from the project instead of using OpenStack directly. I >> was a little turned off from Triple-O when I saw it was source only. The >> feeling being that it is "too green" for our tastes. Which may be >> inaccurate. While I might be convinced to use source, its a much harder sell >> to us currently then using packages. >> > > Kevin, thanks for sharing. I think I understand that it is just a new > way of thinking and that makes it that much harder to consume. > > We have good reasons for not using packages. And we're not just making > this up as a new crazy idea, we're copying what companies like Netflix > have published about running at scale. We need to do a better job at > making sure why we're doing some of the things we're doing. Do you have a link for the publication handy? I know they use a blessed AMI approach. But I'm curious about the "not using packages" part, and the advantages they get from that. All I could find from googling is people trying to install netflix from packages to watch movies :). Their AMI build tool seems to indicate they package their apps: https://github.com/Netflix/aminator As does this presentation: http://www.slideshare.net/garethbowles/building-netflixstreamingwithjenkins-juc -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
- Original Message - > From: "Clint Byrum" > To: "openstack-dev" > Sent: Tuesday, January 7, 2014 3:23:24 PM > Subject: Re: [openstack-dev] [TripleO] Installing from packages in > tripleo-image-elements > > What would be the benefit of using packages? > > We've specifically avoided packages because they complect[1] configuration > and system state management with software delivery. The recent friction > we've seen with MySQL is an example where the packages are not actually > helping us, they're hurting us because they encode too much configuration > instead of just delivering binaries. > > Perhaps those of us who have been involved a bit longer haven't done > a good job of communicating our reasons. I for one believe in the idea > that image based updates eliminate a lot of the entropy that comes along > with a package based updating system. For that reason alone I tend to > look at any packages that deliver configurable software as potentially > dangerous (note that I think they're wonderful for libraries, utilities, > and kernels. :) To be clear James is talking initially about using packages to build images (not update them at runtime). In any case for much the same reason I look at what we do today with the source based pip installs an an entropy bomb: -multiple venvs in the same image -each venv contains its own copies of libraries -everything is compiled on the fly -due to the above ^^ image builds often fail!!! I'd expect that anyone using real packages for OpenStack element has much less entropy on many fronts: -more control over what is installed -faster image build times -less duplication The cost when using real packages is maintaining them. But once you have that (which we will)... you've got a lot more control over things. > > [1] http://www.infoq.com/presentations/Simple-Made-Easy > > Excerpts from James Slagle's message of 2014-01-07 12:01:07 -0800: > > Hi, > > > > I'd like to discuss some possible ways we could install the OpenStack > > components from packages in tripleo-image-elements. As most folks are > > probably aware, there is a "fork" of tripleo-image-elements called > > tripleo-puppet-elements which does install using packages, but it does > > so using Puppet to do the installation and for managing the > > configuration of the installed components. I'd like to kind of set > > that aside for a moment and just discuss how we might support > > installing from packages using tripleo-image-elements directly and not > > using Puppet. > > > > One idea would be to add support for a new type (or likely 2 new > > types: rpm and dpkg) to the source-repositories element. > > source-repositories already knows about the git, tar, and file types, > > so it seems somewhat natural to have additional types for rpm and > > dpkg. > > > > A complication with that approach is that the existing elements assume > > they're setting up everything from source. So, if we take a look at > > the nova element, and specifically install.d/74-nova, that script does > > stuff like install a nova service, adds a nova user, creates needed > > directories, etc. All of that wouldn't need to be done if we were > > installing from rpm or dpkg, b/c presumably the package would take > > care of all that. > > > > We could fix that by making the install.d scripts only run if you're > > installing a component from source. In that sense, it might make > > sense to add a new hook, source-install.d and only run those scripts > > if the type is a source type in the source-repositories configuration. > > We could then have a package-install.d to handle the installation > > from the packages type. The install.d hook could still exist to do > > things that might be common to the 2 methods. > > > > Thoughts on that approach or other ideas? > > > > I'm currently working on a patchset I can submit to help prove it out. > > But, I'd like to start discussion on the approach now to see if there > > are other ideas or major opposition to that approach. > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi > On 7 Jan 2014, at 23:04, Clint Byrum wrote: > > My question still stands, what are the real advantages? So far the only > one that matters to me is "makes it easier for people to think about > using it." If I were to put on my former sysadmin hat, I would always strongly prefer to use packages for things, so I have the weight of the distro vendor behind me (particularly if I only have a few hundred servers and want a 6 month upgrade cadence with easy access to security fixes between upgrades). I'm not necessarily advocating for or against tripleo supporting packages as a source of openstack, but I do think it is likely that some/many users will have their reasons for wanting to leverage our tools without following all of our preferred processes. What I am advocating though, is that if there is a need and someone gives us a patch that satisfies it without hurting our preferred processes, we look favourably on it. I would rather have users on the road to doing things our way, than be immediately turned away or forced into dramatically forking us. Cheers, -- Chris Jones ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
- Original Message - > From: "James Slagle" > To: "OpenStack Development Mailing List (not for usage questions)" > > Sent: Tuesday, January 7, 2014 3:53:57 PM > Subject: Re: [openstack-dev] [TripleO] Installing from packages in > tripleo-image-elements > > On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: > > What would be the benefit of using packages? > > We're building images on top of different distributions of Linux. > Those distributions themselves offer packaged and supported OpenStack > components. So, one benefit is that you'd be using what's "blessed" > by your distro if you chose to. I think that's a farily common way > people are going to be used to installing components. The OpenStack > Installation guide says to install from packages, fwiw. > > > We've specifically avoided packages because they complect[1] configuration > > and system state management with software delivery. The recent friction > > we've seen with MySQL is an example where the packages are not actually > > helping us, they're hurting us because they encode too much configuration > > instead of just delivering binaries. > > We're trying to do something fairly specific with the read only / > partition. You're right, most packages aren't going to handle that > well. So, yes you have a point from that perspective. > > However, there are many examples of when packages help you. > Dependency resolution, version compatibility, methods of verification, > knowing what's installed, etc. I don't think that's really an > argument or discussion worth having, because you either want to use > packages or you want to build it all from source. There are > advantages/disadvantages to both methods, and what I'm proposing is > that we support both methods, and not require everyone to only be able > to install from source. I think James gives a nice summary here. The fact is some people do want to use packages with the OpenStack image elements and we should support them. And from the sounds of it the implementation details aren't going to cost that much. The benefit we get from this is that we as a community can focus in on making a single set of tripleo-image-elements rock solid for both packages and source installs. > > > Perhaps those of us who have been involved a bit longer haven't done > > a good job of communicating our reasons. I for one believe in the idea > > that image based updates eliminate a lot of the entropy that comes along > > with a package based updating system. For that reason alone I tend to > > look at any packages that deliver configurable software as potentially > > dangerous (note that I think they're wonderful for libraries, utilities, > > and kernels. :) > > Using packages wouldn't prevent you from using the image based update > mechanism. Anecdotally, I think image based updates could be a bit > heavy handed for something like picking up a quick security or bug fix > or the like. That would be a scenario where a package update could > really be handy. Especially if someone else (e.g., your distro) is > maintaining the package for you. > > For this proposal though, I was only talking about installation of the > components at image build time. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
- Original Message - > From: "James Slagle" > To: "OpenStack Development Mailing List" > Sent: Tuesday, January 7, 2014 3:01:07 PM > Subject: [openstack-dev] [TripleO] Installing from packages in > tripleo-image-elements > > Hi, > > I'd like to discuss some possible ways we could install the OpenStack > components from packages in tripleo-image-elements. As most folks are > probably aware, there is a "fork" of tripleo-image-elements called > tripleo-puppet-elements which does install using packages, but it does > so using Puppet to do the installation and for managing the > configuration of the installed components. I'd like to kind of set > that aside for a moment and just discuss how we might support > installing from packages using tripleo-image-elements directly and not > using Puppet. I very much support having the option to use real packages within the tripleo-image-elements. Given we already use packages for some elements like Rabbit/MySQL/etc... supporting the option to use standard distribution packages for the OpenStack elements makes a lot of sense to me. > > One idea would be to add support for a new type (or likely 2 new > types: rpm and dpkg) to the source-repositories element. > source-repositories already knows about the git, tar, and file types, > so it seems somewhat natural to have additional types for rpm and > dpkg. > > A complication with that approach is that the existing elements assume > they're setting up everything from source. So, if we take a look at > the nova element, and specifically install.d/74-nova, that script does > stuff like install a nova service, adds a nova user, creates needed > directories, etc. All of that wouldn't need to be done if we were > installing from rpm or dpkg, b/c presumably the package would take > care of all that. > > We could fix that by making the install.d scripts only run if you're > installing a component from source. In that sense, it might make > sense to add a new hook, source-install.d and only run those scripts > if the type is a source type in the source-repositories configuration. > We could then have a package-install.d to handle the installation > from the packages type. The install.d hook could still exist to do > things that might be common to the 2 methods. > > Thoughts on that approach or other ideas? > > I'm currently working on a patchset I can submit to help prove it out. > But, I'd like to start discussion on the approach now to see if there > are other ideas or major opposition to that approach. This seems like a reasonable first stab at it. Can't wait to see what you come up with. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 6:12 PM, Clint Byrum wrote: > Excerpts from Fox, Kevin M's message of 2014-01-07 13:11:13 -0800: >> I was going to stay silent on this one, but since you asked... >> >> /me puts his customer hat on >> >> We source OpenStack from RDO for the packages and additional integration >> testing that comes from the project instead of using OpenStack directly. I >> was a little turned off from Triple-O when I saw it was source only. The >> feeling being that it is "too green" for our tastes. Which may be >> inaccurate. While I might be convinced to use source, its a much harder sell >> to us currently then using packages. >> > > Kevin, thanks for sharing. I think I understand that it is just a new > way of thinking and that makes it that much harder to consume. > > We have good reasons for not using packages. And we're not just making > this up as a new crazy idea, we're copying what companies like Netflix > have published about running at scale. We need to do a better job at > making sure why we're doing some of the things we're doing. Do you have a link for the publication handy? I know they use a blessed AMI approach. But I'm curious about the "not using packages" part, and the advantages they get from that. All I could find from googling is people trying to install netflix from packages to watch movies :). Their AMI build tool seems to indicate they package their apps: https://github.com/Netflix/aminator As does this presentation: http://www.slideshare.net/garethbowles/building-netflixstreamingwithjenkins-juc -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from James Slagle's message of 2014-01-07 15:03:33 -0800: > On Tue, Jan 7, 2014 at 5:18 PM, Clint Byrum wrote: > > Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800: > >> On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: > >> > What would be the benefit of using packages? > >> > >> We're building images on top of different distributions of Linux. > >> Those distributions themselves offer packaged and supported OpenStack > >> components. So, one benefit is that you'd be using what's "blessed" > >> by your distro if you chose to. I think that's a farily common way > >> people are going to be used to installing components. The OpenStack > >> Installation guide says to install from packages, fwiw. > >> > > > > Indeed, this is how many people deploy traditional applications. > > > > However, what we're doing is intended to be a real, consumable deployment > > of OpenStack. Specifically one that is in the gate and scales out to > > any reasonable production load necessary. > > > > One problem with scaling out to many nodes is that the traditional > > application deployment patterns introduce too much entropy. This is really > > hard to patch out later. We are designing the software distribution system > > to avoid those problems from the beginning. Packages do the opposite, > > and encourage entropy by promising to try and update software with > > minimal invasion, thus enabling users to introduce one-off machines. > > This sounds more like an argument for a systems management approach > vs. installation. I realize there's a big relation there. However, I > don't think just using an image based system makes the entropy problem > go away. At scale of 10,000 nodes (or more), you could easily have a > proliferation of images both that you've built and that you've > deployed. You're not going to update everything at once. Nor, do I > think you would only ever have 2 images running, for your latest > version N, and N-1.. You're going to have several different deployed > images running to account for hardware differences, updates that have > not yet been applied, migrations, etc. My point is, the entropy > problem does not go away. Ergo, it's not introduced by packages. > > Certainly it could be made worse by managing packages and their > updates by hand across 10,000 nodes. But again, I don't think anyone > does that, they use a systems management tool that exists to > discourage drift and help with such entropy. > > Likewise, you're not going to be calling "nova rebuild" 10,000 times > manually when you want to do image based updates. You'd likely have > some tool (tuskar, something else, etc) that is going to manage that > for you, and help keep any drift in what images you actually have > deployed in check. > Image proliferation is far easier to measure than package proliferation. But, that is not really the point. The point is that we have a tool for software distribution that fits into our system management approach, and the distro packages do not take that system management approach into account. So if you can't use the distro packages as-is, I'm questioning what the actual benefit of using them at all is. > > > >> > We've specifically avoided packages because they complect[1] > >> > configuration > >> > and system state management with software delivery. The recent friction > >> > we've seen with MySQL is an example where the packages are not actually > >> > helping us, they're hurting us because they encode too much configuration > >> > instead of just delivering binaries. > >> > >> We're trying to do something fairly specific with the read only / > >> partition. You're right, most packages aren't going to handle that > >> well. So, yes you have a point from that perspective. > >> > > > > Readonly / is a really important feature of the deployment we're aiming > > at. Doing it with packages is quite possible. My point in asking why > > bother with packages is that when you have an entire image that has been > > verified and is known to work, what advantage does having a package for > > everything actually bring. > > Because distro packages are known to work, and thus you get higher > confidence from any image constructed from said packages. At least, I > would, as opposed to installing from source (or "from git" as you say > below :). It's the same reason I want to use a packaged kernel > instead of compiling it from source. The benefit of the package is > not just in the compiling. It's in the known good version and > compatibility with other known good versions I want to use. > I would disagree with "known to work". They are known to have been tested at some level. But IMO "known to work" requires testing _with your workload_. Since you have to test your workload, why bother with the distro packages when you can get the upstream software and testing suite directly. > Am I going to implicitly trust any packages blindly or completely? Of > course no
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
One of the major features using a distro over upstream gives is integration. rhel6 behaves differently then ubuntu 13.10. Sometimes it takes a while to fix upstream for a given distro, and even then it may not even be accepted because "the distro's too old, go away". Packages allow a distro to make sure all the pieces can work together properly. And quickly patch things when needed. Its kind of a fork but not quite the same thing. The distro integrates not just OpenStack, but all its dependencies and their dependencies all the way up. For example there can be subtle issues if neutron, open vswitch and the kernel are out of sync. It is the integration that folks like about distro's. "I can trust that since it came from X, all the pieces should work together, or I know who to call". The only way I can think of to get the same stability out of just source is for Triple-O to provide a whole source distro and test it, itself. More work then it probably wants to do. Or pick just one distro and support source only on that. Though if you pick the wrong distro, then you get into trust issues and religious wars. :/ Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Tuesday, January 07, 2014 3:04 PM To: openstack-dev Subject: Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements Excerpts from Chris Jones's message of 2014-01-07 14:43:31 -0800: > Hi > > > On 7 Jan 2014, at 22:18, Clint Byrum wrote: > > Packages do the opposite, > > and encourage entropy by promising to try and update software > > Building with packages doesn't require updating running systems with packages > and more than building with git requires updating running systems with git > pull. > One can simply build (and test!) a new image with updated packages and > rebuild/takeover nodes. > Indeed, however one can _more_ simply build an image without package tooling... and they will be more similar across multiple platforms. My question still stands, what are the real advantages? So far the only one that matters to me is "makes it easier for people to think about using it." ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 6:04 PM, Clint Byrum wrote: > Excerpts from Chris Jones's message of 2014-01-07 14:43:31 -0800: >> Hi >> >> > On 7 Jan 2014, at 22:18, Clint Byrum wrote: >> > Packages do the opposite, >> > and encourage entropy by promising to try and update software >> >> Building with packages doesn't require updating running systems with >> packages and more than building with git requires updating running systems >> with git pull. >> One can simply build (and test!) a new image with updated packages and >> rebuild/takeover nodes. >> > > Indeed, however one can _more_ simply build an image without package > tooling... and they will be more similar across multiple platforms. > > My question still stands, what are the real advantages? So far the only > one that matters to me is "makes it easier for people to think about > using it." I'm reminded of when I first started looking at TripleO there were a few issues with installing from git (I'll say that from now on :) related to all the python distribute -> setuptools migration. Things like if you're base cloud image had the wrong version of pip you couldn't migrate to setuptools cleanly. Then you had to run the setuptools update twice, once to get the distribute legacy wrapper and then again to latest setuptools. If I recall there were other problems with virtualenv incompatibilities as well. Arguably, installing from packages would have made that easier and less complex. Sure, the crux of the problem was likely that versions in the distro were too old and they needed to be updated. But unless we take on building the whole OS from source/git/whatever every time, we're always going to have that issue. So, an additional benefit of packages is that you can install a known good version of an OpenStack component that is known to work with the versions of dependent software you already have installed. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Fox, Kevin M's message of 2014-01-07 13:11:13 -0800: > I was going to stay silent on this one, but since you asked... > > /me puts his customer hat on > > We source OpenStack from RDO for the packages and additional integration > testing that comes from the project instead of using OpenStack directly. I > was a little turned off from Triple-O when I saw it was source only. The > feeling being that it is "too green" for our tastes. Which may be inaccurate. > While I might be convinced to use source, its a much harder sell to us > currently then using packages. > Kevin, thanks for sharing. I think I understand that it is just a new way of thinking and that makes it that much harder to consume. We have good reasons for not using packages. And we're not just making this up as a new crazy idea, we're copying what companies like Netflix have published about running at scale. We need to do a better job at making sure why we're doing some of the things we're doing. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from Chris Jones's message of 2014-01-07 14:43:31 -0800: > Hi > > > On 7 Jan 2014, at 22:18, Clint Byrum wrote: > > Packages do the opposite, > > and encourage entropy by promising to try and update software > > Building with packages doesn't require updating running systems with packages > and more than building with git requires updating running systems with git > pull. > One can simply build (and test!) a new image with updated packages and > rebuild/takeover nodes. > Indeed, however one can _more_ simply build an image without package tooling... and they will be more similar across multiple platforms. My question still stands, what are the real advantages? So far the only one that matters to me is "makes it easier for people to think about using it." ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 5:18 PM, Clint Byrum wrote: > Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800: >> On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: >> > What would be the benefit of using packages? >> >> We're building images on top of different distributions of Linux. >> Those distributions themselves offer packaged and supported OpenStack >> components. So, one benefit is that you'd be using what's "blessed" >> by your distro if you chose to. I think that's a farily common way >> people are going to be used to installing components. The OpenStack >> Installation guide says to install from packages, fwiw. >> > > Indeed, this is how many people deploy traditional applications. > > However, what we're doing is intended to be a real, consumable deployment > of OpenStack. Specifically one that is in the gate and scales out to > any reasonable production load necessary. > > One problem with scaling out to many nodes is that the traditional > application deployment patterns introduce too much entropy. This is really > hard to patch out later. We are designing the software distribution system > to avoid those problems from the beginning. Packages do the opposite, > and encourage entropy by promising to try and update software with > minimal invasion, thus enabling users to introduce one-off machines. This sounds more like an argument for a systems management approach vs. installation. I realize there's a big relation there. However, I don't think just using an image based system makes the entropy problem go away. At scale of 10,000 nodes (or more), you could easily have a proliferation of images both that you've built and that you've deployed. You're not going to update everything at once. Nor, do I think you would only ever have 2 images running, for your latest version N, and N-1.. You're going to have several different deployed images running to account for hardware differences, updates that have not yet been applied, migrations, etc. My point is, the entropy problem does not go away. Ergo, it's not introduced by packages. Certainly it could be made worse by managing packages and their updates by hand across 10,000 nodes. But again, I don't think anyone does that, they use a systems management tool that exists to discourage drift and help with such entropy. Likewise, you're not going to be calling "nova rebuild" 10,000 times manually when you want to do image based updates. You'd likely have some tool (tuskar, something else, etc) that is going to manage that for you, and help keep any drift in what images you actually have deployed in check. > >> > We've specifically avoided packages because they complect[1] configuration >> > and system state management with software delivery. The recent friction >> > we've seen with MySQL is an example where the packages are not actually >> > helping us, they're hurting us because they encode too much configuration >> > instead of just delivering binaries. >> >> We're trying to do something fairly specific with the read only / >> partition. You're right, most packages aren't going to handle that >> well. So, yes you have a point from that perspective. >> > > Readonly / is a really important feature of the deployment we're aiming > at. Doing it with packages is quite possible. My point in asking why > bother with packages is that when you have an entire image that has been > verified and is known to work, what advantage does having a package for > everything actually bring. Because distro packages are known to work, and thus you get higher confidence from any image constructed from said packages. At least, I would, as opposed to installing from source (or "from git" as you say below :). It's the same reason I want to use a packaged kernel instead of compiling it from source. The benefit of the package is not just in the compiling. It's in the known good version and compatibility with other known good versions I want to use. Am I going to implicitly trust any packages blindly or completely? Of course not. But, there is some confidence there in that the distro has done some testing and said these versions are compatible, etc. > >> However, there are many examples of when packages help you. >> Dependency resolution, version compatibility, methods of verification, >> knowing what's installed, etc. I don't think that's really an >> argument or discussion worth having, because you either want to use >> packages or you want to build it all from source. There are >> advantages/disadvantages to both methods, and what I'm proposing is >> that we support both methods, and not require everyone to only be able >> to install from source. >> > > "Install from source" is probably not the right way to put this. We're > installing the virtualenvs from tarballs downloaded from pypi. We're > also installing 99.9% python, so we're not really going "from source", > we're just going "from git". Yes, "from source" basically means "from git"
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi > On 7 Jan 2014, at 22:18, Clint Byrum wrote: > Packages do the opposite, > and encourage entropy by promising to try and update software Building with packages doesn't require updating running systems with packages and more than building with git requires updating running systems with git pull. One can simply build (and test!) a new image with updated packages and rebuild/takeover nodes. Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi > On 7 Jan 2014, at 21:17, James Slagle wrote: > > Could you expand on this a bit? I'm not sure what inconsistency > you're referring to. That multiple repos work, but the docs don't say so, and the DIB_REPO foo doesn't support multiple repos. > I wonder if we could have a global build option as well that said to use > packages or source Definitely. Maybe DIB_PREFER_ORIGIN? > I feel that not offering a choice will only turn people off from using > t-i-e. +1 (even if I wish that wasn't the case!) Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800: > On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: > > What would be the benefit of using packages? > > We're building images on top of different distributions of Linux. > Those distributions themselves offer packaged and supported OpenStack > components. So, one benefit is that you'd be using what's "blessed" > by your distro if you chose to. I think that's a farily common way > people are going to be used to installing components. The OpenStack > Installation guide says to install from packages, fwiw. > Indeed, this is how many people deploy traditional applications. However, what we're doing is intended to be a real, consumable deployment of OpenStack. Specifically one that is in the gate and scales out to any reasonable production load necessary. One problem with scaling out to many nodes is that the traditional application deployment patterns introduce too much entropy. This is really hard to patch out later. We are designing the software distribution system to avoid those problems from the beginning. Packages do the opposite, and encourage entropy by promising to try and update software with minimal invasion, thus enabling users to introduce one-off machines. > > We've specifically avoided packages because they complect[1] configuration > > and system state management with software delivery. The recent friction > > we've seen with MySQL is an example where the packages are not actually > > helping us, they're hurting us because they encode too much configuration > > instead of just delivering binaries. > > We're trying to do something fairly specific with the read only / > partition. You're right, most packages aren't going to handle that > well. So, yes you have a point from that perspective. > Readonly / is a really important feature of the deployment we're aiming at. Doing it with packages is quite possible. My point in asking why bother with packages is that when you have an entire image that has been verified and is known to work, what advantage does having a package for everything actually bring. > However, there are many examples of when packages help you. > Dependency resolution, version compatibility, methods of verification, > knowing what's installed, etc. I don't think that's really an > argument or discussion worth having, because you either want to use > packages or you want to build it all from source. There are > advantages/disadvantages to both methods, and what I'm proposing is > that we support both methods, and not require everyone to only be able > to install from source. > "Install from source" is probably not the right way to put this. We're installing the virtualenvs from tarballs downloaded from pypi. We're also installing 99.9% python, so we're not really going "from source", we're just going "from git". But anyway, I see your point and will capitulate that it is less weird for people and thus may make the pill a little easier to swallow. But if I could have it my way, I'd suggest that the packages be built to mirror the structure of the element end-products as much as possible so that they can be used with minimal change to elements. > > Perhaps those of us who have been involved a bit longer haven't done > > a good job of communicating our reasons. I for one believe in the idea > > that image based updates eliminate a lot of the entropy that comes along > > with a package based updating system. For that reason alone I tend to > > look at any packages that deliver configurable software as potentially > > dangerous (note that I think they're wonderful for libraries, utilities, > > and kernels. :) > > Using packages wouldn't prevent you from using the image based update > mechanism. Anecdotally, I think image based updates could be a bit > heavy handed for something like picking up a quick security or bug fix > or the like. That would be a scenario where a package update could > really be handy. Especially if someone else (e.g., your distro) is > maintaining the package for you. > > For this proposal though, I was only talking about installation of the > components at image build time. > The entire point of image based updates is that they are heavy handed. The problem we're trying to solve is that you have a data center of (n) machines and you don't want (n) unique sets of software, where each machine might have some hot fixes and not others. At 1000 machines it becomes critical. At 1 machines, the entropy matrix starts to get scary. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 3:48 PM, Chris Jones wrote: > Hi > > Assuming we want to do this, but not necessarily agreeing that we do want to, > I would suggest: > > 1) I think it would be nice if we could avoid separate dpkg/rpm types by > having a package type and reusing the package map facility. Indeed, I'd like to see one package type as well. I think we could start with that route, and only split it out if there was a proven technical need. > 2) Clear up the source-repositories inconsistency by making it clear that > multiple repositories of the same type do not work in > source-repositories-nova (this would be a behaviour change, but would mesh > more closely with the docs, and would require refactoring the 4 elements we > ship atm with multiple git repos listed) Could you expand on this a bit? I'm not sure what inconsistency you're referring to. > 3) extend arg_to_element to parse element names like "nova/package", > "nova/tar", "nova/file" and "nova/source" (defaulting to source), storing the > choice for later. > > 4) When processing the nova element, apply only the appropriate entry in > source-repositories-nova > > 5) Keep install.d as-is and make the scripts be aware of the previously > stored choice of element origin in the elements (as they add support for a > package origin) > > 6) Probably rename source-repositories to something more appropriate. All good ideas. I like the mechanism to specify the type as well. I wonder if we could have a global build option as well that said to use packages or source, or whatever, for all components that support that type. That way you wouldn't have to specify each individually. > As for whether we should do this or not... like Clint I want to say no, but > I'm also worried about people forking t-i-e and not pushing their > fixes/improvements and new elements back up to us because we're too diverged. I feel that not offering a choice will only turn people off from using t-i-e. Only offering an install from source option is not likely to cause large groups of people to suddenly decide that only installing from source is the way to go and then start using t-i-e exclusively. So, that's why I'd really like to see support for packages in the main repo itself. > If this is a real customer need, I would come down in favour of doing it if > the cost of the above implementation (or an alternate one) isn't too high. +1. Installing from source (master) would still be the default. And any implementations that allowed something different would have to not disrupt that. Similar to how we've added new install options in the paste (source-repositories, tar, etc) and have kept disruptions to a minimum. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
I was going to stay silent on this one, but since you asked... /me puts his customer hat on We source OpenStack from RDO for the packages and additional integration testing that comes from the project instead of using OpenStack directly. I was a little turned off from Triple-O when I saw it was source only. The feeling being that it is "too green" for our tastes. Which may be inaccurate. While I might be convinced to use source, its a much harder sell to us currently then using packages. /me takes of his customer hat. Thanks again for all the hard work on Triple-O. Its looking great, and I hope I get the chance to use it soon. Thanks, Kevin From: Chris Jones [c...@tenshu.net] Sent: Tuesday, January 07, 2014 12:48 PM To: OpenStack Development Mailing List (not for usage questions) Cc: OpenStack Development Mailing List Subject: Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements Hi Assuming we want to do this, but not necessarily agreeing that we do want to, I would suggest: 1) I think it would be nice if we could avoid separate dpkg/rpm types by having a package type and reusing the package map facility. 2) Clear up the source-repositories inconsistency by making it clear that multiple repositories of the same type do not work in source-repositories-nova (this would be a behaviour change, but would mesh more closely with the docs, and would require refactoring the 4 elements we ship atm with multiple git repos listed) 3) extend arg_to_element to parse element names like "nova/package", "nova/tar", "nova/file" and "nova/source" (defaulting to source), storing the choice for later. 4) When processing the nova element, apply only the appropriate entry in source-repositories-nova 5) Keep install.d as-is and make the scripts be aware of the previously stored choice of element origin in the elements (as they add support for a package origin) 6) Probably rename source-repositories to something more appropriate. As for whether we should do this or not... like Clint I want to say no, but I'm also worried about people forking t-i-e and not pushing their fixes/improvements and new elements back up to us because we're too diverged. If this is a real customer need, I would come down in favour of doing it if the cost of the above implementation (or an alternate one) isn't too high. Cheers, -- Chris Jones > On 7 Jan 2014, at 20:01, James Slagle wrote: > > Hi, > > I'd like to discuss some possible ways we could install the OpenStack > components from packages in tripleo-image-elements. As most folks are > probably aware, there is a "fork" of tripleo-image-elements called > tripleo-puppet-elements which does install using packages, but it does > so using Puppet to do the installation and for managing the > configuration of the installed components. I'd like to kind of set > that aside for a moment and just discuss how we might support > installing from packages using tripleo-image-elements directly and not > using Puppet. > > One idea would be to add support for a new type (or likely 2 new > types: rpm and dpkg) to the source-repositories element. > source-repositories already knows about the git, tar, and file types, > so it seems somewhat natural to have additional types for rpm and > dpkg. > > A complication with that approach is that the existing elements assume > they're setting up everything from source. So, if we take a look at > the nova element, and specifically install.d/74-nova, that script does > stuff like install a nova service, adds a nova user, creates needed > directories, etc. All of that wouldn't need to be done if we were > installing from rpm or dpkg, b/c presumably the package would take > care of all that. > > We could fix that by making the install.d scripts only run if you're > installing a component from source. In that sense, it might make > sense to add a new hook, source-install.d and only run those scripts > if the type is a source type in the source-repositories configuration. > We could then have a package-install.d to handle the installation > from the packages type. The install.d hook could still exist to do > things that might be common to the 2 methods. > > Thoughts on that approach or other ideas? > > I'm currently working on a patchset I can submit to help prove it out. > But, I'd like to start discussion on the approach now to see if there > are other ideas or major opposition to that approach. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi (FWIW I suggested using the element arguments like "nova/package" to avoid a huge and crazy environment by using DIB_REPO foo for every element) Cheers, -- Chris Jones > On 7 Jan 2014, at 20:32, James Slagle wrote: > >> On Tue, Jan 7, 2014 at 3:22 PM, Fox, Kevin M wrote: >> Sounds very useful. Would there be a diskimage-builder flag then to say you >> prefer packages over source? Would it fall back to source if you specified >> packages and there were only source-install.d for a given element? > > Yes, you could pick which you wanted via environment variables. > Similar to the way you can pick if you want git head, a specific > gerrit review, or a released tarball today via $DIB_REPOTYPE_, > etc. See: > https://github.com/openstack/diskimage-builder/blob/master/elements/source-repositories/README.md > for more info about that. > > If you specified something that didn't exist, it should probably fail > with an error. The default behavior would still be installing from > git master source if you specified nothing though. > > >> >> Thanks, >> Kevin >> >> From: James Slagle [james.sla...@gmail.com] >> Sent: Tuesday, January 07, 2014 12:01 PM >> To: OpenStack Development Mailing List >> Subject: [openstack-dev] [TripleO] Installing from packages in >> tripleo-image-elements >> >> Hi, >> >> I'd like to discuss some possible ways we could install the OpenStack >> components from packages in tripleo-image-elements. As most folks are >> probably aware, there is a "fork" of tripleo-image-elements called >> tripleo-puppet-elements which does install using packages, but it does >> so using Puppet to do the installation and for managing the >> configuration of the installed components. I'd like to kind of set >> that aside for a moment and just discuss how we might support >> installing from packages using tripleo-image-elements directly and not >> using Puppet. >> >> One idea would be to add support for a new type (or likely 2 new >> types: rpm and dpkg) to the source-repositories element. >> source-repositories already knows about the git, tar, and file types, >> so it seems somewhat natural to have additional types for rpm and >> dpkg. >> >> A complication with that approach is that the existing elements assume >> they're setting up everything from source. So, if we take a look at >> the nova element, and specifically install.d/74-nova, that script does >> stuff like install a nova service, adds a nova user, creates needed >> directories, etc. All of that wouldn't need to be done if we were >> installing from rpm or dpkg, b/c presumably the package would take >> care of all that. >> >> We could fix that by making the install.d scripts only run if you're >> installing a component from source. In that sense, it might make >> sense to add a new hook, source-install.d and only run those scripts >> if the type is a source type in the source-repositories configuration. >> We could then have a package-install.d to handle the installation >> from the packages type. The install.d hook could still exist to do >> things that might be common to the 2 methods. >> >> Thoughts on that approach or other ideas? >> >> I'm currently working on a patchset I can submit to help prove it out. >> But, I'd like to start discussion on the approach now to see if there >> are other ideas or major opposition to that approach. >> >> -- >> -- James Slagle >> -- >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum wrote: > What would be the benefit of using packages? We're building images on top of different distributions of Linux. Those distributions themselves offer packaged and supported OpenStack components. So, one benefit is that you'd be using what's "blessed" by your distro if you chose to. I think that's a farily common way people are going to be used to installing components. The OpenStack Installation guide says to install from packages, fwiw. > We've specifically avoided packages because they complect[1] configuration > and system state management with software delivery. The recent friction > we've seen with MySQL is an example where the packages are not actually > helping us, they're hurting us because they encode too much configuration > instead of just delivering binaries. We're trying to do something fairly specific with the read only / partition. You're right, most packages aren't going to handle that well. So, yes you have a point from that perspective. However, there are many examples of when packages help you. Dependency resolution, version compatibility, methods of verification, knowing what's installed, etc. I don't think that's really an argument or discussion worth having, because you either want to use packages or you want to build it all from source. There are advantages/disadvantages to both methods, and what I'm proposing is that we support both methods, and not require everyone to only be able to install from source. > Perhaps those of us who have been involved a bit longer haven't done > a good job of communicating our reasons. I for one believe in the idea > that image based updates eliminate a lot of the entropy that comes along > with a package based updating system. For that reason alone I tend to > look at any packages that deliver configurable software as potentially > dangerous (note that I think they're wonderful for libraries, utilities, > and kernels. :) Using packages wouldn't prevent you from using the image based update mechanism. Anecdotally, I think image based updates could be a bit heavy handed for something like picking up a quick security or bug fix or the like. That would be a scenario where a package update could really be handy. Especially if someone else (e.g., your distro) is maintaining the package for you. For this proposal though, I was only talking about installation of the components at image build time. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi Assuming we want to do this, but not necessarily agreeing that we do want to, I would suggest: 1) I think it would be nice if we could avoid separate dpkg/rpm types by having a package type and reusing the package map facility. 2) Clear up the source-repositories inconsistency by making it clear that multiple repositories of the same type do not work in source-repositories-nova (this would be a behaviour change, but would mesh more closely with the docs, and would require refactoring the 4 elements we ship atm with multiple git repos listed) 3) extend arg_to_element to parse element names like "nova/package", "nova/tar", "nova/file" and "nova/source" (defaulting to source), storing the choice for later. 4) When processing the nova element, apply only the appropriate entry in source-repositories-nova 5) Keep install.d as-is and make the scripts be aware of the previously stored choice of element origin in the elements (as they add support for a package origin) 6) Probably rename source-repositories to something more appropriate. As for whether we should do this or not... like Clint I want to say no, but I'm also worried about people forking t-i-e and not pushing their fixes/improvements and new elements back up to us because we're too diverged. If this is a real customer need, I would come down in favour of doing it if the cost of the above implementation (or an alternate one) isn't too high. Cheers, -- Chris Jones > On 7 Jan 2014, at 20:01, James Slagle wrote: > > Hi, > > I'd like to discuss some possible ways we could install the OpenStack > components from packages in tripleo-image-elements. As most folks are > probably aware, there is a "fork" of tripleo-image-elements called > tripleo-puppet-elements which does install using packages, but it does > so using Puppet to do the installation and for managing the > configuration of the installed components. I'd like to kind of set > that aside for a moment and just discuss how we might support > installing from packages using tripleo-image-elements directly and not > using Puppet. > > One idea would be to add support for a new type (or likely 2 new > types: rpm and dpkg) to the source-repositories element. > source-repositories already knows about the git, tar, and file types, > so it seems somewhat natural to have additional types for rpm and > dpkg. > > A complication with that approach is that the existing elements assume > they're setting up everything from source. So, if we take a look at > the nova element, and specifically install.d/74-nova, that script does > stuff like install a nova service, adds a nova user, creates needed > directories, etc. All of that wouldn't need to be done if we were > installing from rpm or dpkg, b/c presumably the package would take > care of all that. > > We could fix that by making the install.d scripts only run if you're > installing a component from source. In that sense, it might make > sense to add a new hook, source-install.d and only run those scripts > if the type is a source type in the source-repositories configuration. > We could then have a package-install.d to handle the installation > from the packages type. The install.d hook could still exist to do > things that might be common to the 2 methods. > > Thoughts on that approach or other ideas? > > I'm currently working on a patchset I can submit to help prove it out. > But, I'd like to start discussion on the approach now to see if there > are other ideas or major opposition to that approach. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi My guess for the easiest answer to that: distro vendor support. Cheers, -- Chris Jones > On 7 Jan 2014, at 20:23, Clint Byrum wrote: > > What would be the benefit of using packages? > > We've specifically avoided packages because they complect[1] configuration > and system state management with software delivery. The recent friction > we've seen with MySQL is an example where the packages are not actually > helping us, they're hurting us because they encode too much configuration > instead of just delivering binaries. > > Perhaps those of us who have been involved a bit longer haven't done > a good job of communicating our reasons. I for one believe in the idea > that image based updates eliminate a lot of the entropy that comes along > with a package based updating system. For that reason alone I tend to > look at any packages that deliver configurable software as potentially > dangerous (note that I think they're wonderful for libraries, utilities, > and kernels. :) > > [1] http://www.infoq.com/presentations/Simple-Made-Easy > > Excerpts from James Slagle's message of 2014-01-07 12:01:07 -0800: >> Hi, >> >> I'd like to discuss some possible ways we could install the OpenStack >> components from packages in tripleo-image-elements. As most folks are >> probably aware, there is a "fork" of tripleo-image-elements called >> tripleo-puppet-elements which does install using packages, but it does >> so using Puppet to do the installation and for managing the >> configuration of the installed components. I'd like to kind of set >> that aside for a moment and just discuss how we might support >> installing from packages using tripleo-image-elements directly and not >> using Puppet. >> >> One idea would be to add support for a new type (or likely 2 new >> types: rpm and dpkg) to the source-repositories element. >> source-repositories already knows about the git, tar, and file types, >> so it seems somewhat natural to have additional types for rpm and >> dpkg. >> >> A complication with that approach is that the existing elements assume >> they're setting up everything from source. So, if we take a look at >> the nova element, and specifically install.d/74-nova, that script does >> stuff like install a nova service, adds a nova user, creates needed >> directories, etc. All of that wouldn't need to be done if we were >> installing from rpm or dpkg, b/c presumably the package would take >> care of all that. >> >> We could fix that by making the install.d scripts only run if you're >> installing a component from source. In that sense, it might make >> sense to add a new hook, source-install.d and only run those scripts >> if the type is a source type in the source-repositories configuration. >> We could then have a package-install.d to handle the installation >> from the packages type. The install.d hook could still exist to do >> things that might be common to the 2 methods. >> >> Thoughts on that approach or other ideas? >> >> I'm currently working on a patchset I can submit to help prove it out. >> But, I'd like to start discussion on the approach now to see if there >> are other ideas or major opposition to that approach. >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
On Tue, Jan 7, 2014 at 3:22 PM, Fox, Kevin M wrote: > Sounds very useful. Would there be a diskimage-builder flag then to say you > prefer packages over source? Would it fall back to source if you specified > packages and there were only source-install.d for a given element? Yes, you could pick which you wanted via environment variables. Similar to the way you can pick if you want git head, a specific gerrit review, or a released tarball today via $DIB_REPOTYPE_, etc. See: https://github.com/openstack/diskimage-builder/blob/master/elements/source-repositories/README.md for more info about that. If you specified something that didn't exist, it should probably fail with an error. The default behavior would still be installing from git master source if you specified nothing though. > > Thanks, > Kevin > > From: James Slagle [james.sla...@gmail.com] > Sent: Tuesday, January 07, 2014 12:01 PM > To: OpenStack Development Mailing List > Subject: [openstack-dev] [TripleO] Installing from packages in > tripleo-image-elements > > Hi, > > I'd like to discuss some possible ways we could install the OpenStack > components from packages in tripleo-image-elements. As most folks are > probably aware, there is a "fork" of tripleo-image-elements called > tripleo-puppet-elements which does install using packages, but it does > so using Puppet to do the installation and for managing the > configuration of the installed components. I'd like to kind of set > that aside for a moment and just discuss how we might support > installing from packages using tripleo-image-elements directly and not > using Puppet. > > One idea would be to add support for a new type (or likely 2 new > types: rpm and dpkg) to the source-repositories element. > source-repositories already knows about the git, tar, and file types, > so it seems somewhat natural to have additional types for rpm and > dpkg. > > A complication with that approach is that the existing elements assume > they're setting up everything from source. So, if we take a look at > the nova element, and specifically install.d/74-nova, that script does > stuff like install a nova service, adds a nova user, creates needed > directories, etc. All of that wouldn't need to be done if we were > installing from rpm or dpkg, b/c presumably the package would take > care of all that. > > We could fix that by making the install.d scripts only run if you're > installing a component from source. In that sense, it might make > sense to add a new hook, source-install.d and only run those scripts > if the type is a source type in the source-repositories configuration. > We could then have a package-install.d to handle the installation > from the packages type. The install.d hook could still exist to do > things that might be common to the 2 methods. > > Thoughts on that approach or other ideas? > > I'm currently working on a patchset I can submit to help prove it out. > But, I'd like to start discussion on the approach now to see if there > are other ideas or major opposition to that approach. > > -- > -- James Slagle > -- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
What would be the benefit of using packages? We've specifically avoided packages because they complect[1] configuration and system state management with software delivery. The recent friction we've seen with MySQL is an example where the packages are not actually helping us, they're hurting us because they encode too much configuration instead of just delivering binaries. Perhaps those of us who have been involved a bit longer haven't done a good job of communicating our reasons. I for one believe in the idea that image based updates eliminate a lot of the entropy that comes along with a package based updating system. For that reason alone I tend to look at any packages that deliver configurable software as potentially dangerous (note that I think they're wonderful for libraries, utilities, and kernels. :) [1] http://www.infoq.com/presentations/Simple-Made-Easy Excerpts from James Slagle's message of 2014-01-07 12:01:07 -0800: > Hi, > > I'd like to discuss some possible ways we could install the OpenStack > components from packages in tripleo-image-elements. As most folks are > probably aware, there is a "fork" of tripleo-image-elements called > tripleo-puppet-elements which does install using packages, but it does > so using Puppet to do the installation and for managing the > configuration of the installed components. I'd like to kind of set > that aside for a moment and just discuss how we might support > installing from packages using tripleo-image-elements directly and not > using Puppet. > > One idea would be to add support for a new type (or likely 2 new > types: rpm and dpkg) to the source-repositories element. > source-repositories already knows about the git, tar, and file types, > so it seems somewhat natural to have additional types for rpm and > dpkg. > > A complication with that approach is that the existing elements assume > they're setting up everything from source. So, if we take a look at > the nova element, and specifically install.d/74-nova, that script does > stuff like install a nova service, adds a nova user, creates needed > directories, etc. All of that wouldn't need to be done if we were > installing from rpm or dpkg, b/c presumably the package would take > care of all that. > > We could fix that by making the install.d scripts only run if you're > installing a component from source. In that sense, it might make > sense to add a new hook, source-install.d and only run those scripts > if the type is a source type in the source-repositories configuration. > We could then have a package-install.d to handle the installation > from the packages type. The install.d hook could still exist to do > things that might be common to the 2 methods. > > Thoughts on that approach or other ideas? > > I'm currently working on a patchset I can submit to help prove it out. > But, I'd like to start discussion on the approach now to see if there > are other ideas or major opposition to that approach. > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Sounds very useful. Would there be a diskimage-builder flag then to say you prefer packages over source? Would it fall back to source if you specified packages and there were only source-install.d for a given element? Thanks, Kevin From: James Slagle [james.sla...@gmail.com] Sent: Tuesday, January 07, 2014 12:01 PM To: OpenStack Development Mailing List Subject: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements Hi, I'd like to discuss some possible ways we could install the OpenStack components from packages in tripleo-image-elements. As most folks are probably aware, there is a "fork" of tripleo-image-elements called tripleo-puppet-elements which does install using packages, but it does so using Puppet to do the installation and for managing the configuration of the installed components. I'd like to kind of set that aside for a moment and just discuss how we might support installing from packages using tripleo-image-elements directly and not using Puppet. One idea would be to add support for a new type (or likely 2 new types: rpm and dpkg) to the source-repositories element. source-repositories already knows about the git, tar, and file types, so it seems somewhat natural to have additional types for rpm and dpkg. A complication with that approach is that the existing elements assume they're setting up everything from source. So, if we take a look at the nova element, and specifically install.d/74-nova, that script does stuff like install a nova service, adds a nova user, creates needed directories, etc. All of that wouldn't need to be done if we were installing from rpm or dpkg, b/c presumably the package would take care of all that. We could fix that by making the install.d scripts only run if you're installing a component from source. In that sense, it might make sense to add a new hook, source-install.d and only run those scripts if the type is a source type in the source-repositories configuration. We could then have a package-install.d to handle the installation from the packages type. The install.d hook could still exist to do things that might be common to the 2 methods. Thoughts on that approach or other ideas? I'm currently working on a patchset I can submit to help prove it out. But, I'd like to start discussion on the approach now to see if there are other ideas or major opposition to that approach. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] Installing from packages in tripleo-image-elements
Hi, I'd like to discuss some possible ways we could install the OpenStack components from packages in tripleo-image-elements. As most folks are probably aware, there is a "fork" of tripleo-image-elements called tripleo-puppet-elements which does install using packages, but it does so using Puppet to do the installation and for managing the configuration of the installed components. I'd like to kind of set that aside for a moment and just discuss how we might support installing from packages using tripleo-image-elements directly and not using Puppet. One idea would be to add support for a new type (or likely 2 new types: rpm and dpkg) to the source-repositories element. source-repositories already knows about the git, tar, and file types, so it seems somewhat natural to have additional types for rpm and dpkg. A complication with that approach is that the existing elements assume they're setting up everything from source. So, if we take a look at the nova element, and specifically install.d/74-nova, that script does stuff like install a nova service, adds a nova user, creates needed directories, etc. All of that wouldn't need to be done if we were installing from rpm or dpkg, b/c presumably the package would take care of all that. We could fix that by making the install.d scripts only run if you're installing a component from source. In that sense, it might make sense to add a new hook, source-install.d and only run those scripts if the type is a source type in the source-repositories configuration. We could then have a package-install.d to handle the installation from the packages type. The install.d hook could still exist to do things that might be common to the 2 methods. Thoughts on that approach or other ideas? I'm currently working on a patchset I can submit to help prove it out. But, I'd like to start discussion on the approach now to see if there are other ideas or major opposition to that approach. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
