Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Hey everyone! Try-it vs. Very-complex: Separate projects, with links to eachother in the READMEs. Production: I use my chef recipes in production. Some more organizing and some re-thinking has to go into them to work well. I'd like to start that conversation in another thread, after taking a second look at where other folks have gone with the Swift recipes. I'd reference again my blue-sky blog entries from last summer on Swift and Chef. I still run my cluster that way, but what I've learned since then needs to be integrated. Patches welcome! -judd On Feb 28, 2012 5:33 PM, andi abes andi.a...@gmail.com wrote: On Tue, Feb 28, 2012 at 3:21 PM, Jay Pipes jaypi...@gmail.com wrote: cc'ing Maru since his particular cookbook is being discussed here : On 02/28/2012 02:56 PM, andi abes wrote: yes and no One neat feature of chef is it's search capability - being able to query the sever of where other pieces of the puzzle are located, which makes it very convenient for multi-node operations. E.g. for swift there are a few cookbooks floating around where by the rings are constructed by locating all the servers that are tagged as storage nodes (i.e. they have the appropriate role(s) assigned to them. While search is a neat capability, it does make the recipes more complex (recipes are part of cookbooks, that express the operations to be performed). So if the intent is to have the cookbooks serve as an newbie exemplar, showcasing openstack - its probably not a good idea. Other complexities arise when you start dealing with machine variably, that can be easily hidden in SAIO. Using swift as an example - the # and device names of disks. In SAIO, you just create a bunch of loopback devices... (at least the sample deployment docs do). On a more (dare I say) production environment, you'd want to discover what disks are available, and use the appropriate ones. That said - there could be recipes for both SIAO and multi-node. Users would then have to combine and apply the right set. But maybe that's not the full question... maybe a more complete question would be: is this effort geared towards producing deployments that can be considered production ready? I believe most people would answer Yes. The openstack-chef upstream cookbooks should be geared towards production environments. I was just asking if there is the ability to have both production-ready recipes and try this out recipes all in the same repo. Sounds like that's not really a good thing, and if not, we should decide where the appropriate place to put try this out recipes should be? Didn't mean to say that's not a good idea.. (the try-it part), or not possible. My main goal was just to double check that indeed a production deployment is envisioned. A simple option to ease any possible confusion between the different recipes could be to have separate cookbooks- e.g. swift and swfit-saio or maybe openstack and openstack-saio. However, the mention of production raises a few other questions (again.. I think I'm just echoing) - specifically what's the source of the OpenStack bits to be used... ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/27/2012 08:17 PM, Judd Maltin wrote: Sorry I'm late to the game, but my swift chef recipes are on github as well. No worries, better late than never :) Glad you've joined the fun! It's a fork of andi abes's recipes from back in mid 2011. They contain many an ugly hack and some half-baked ideas. BUT, if you want Ubuntu 10.04, munin, rsyslog, slogging and a bunch of other cool stuff, please give them a try. This is also a good impetus for me updating my swift repo. https://github.com/voxeldotnet/openstack-swift-chef Interesting. Would you mind doing a code review on Mary Newby's Swift All in One cookbook? Could sure use your experience :) https://review.openstack.org/#change,3613 Might be a good dip into the Gerrit world for ya, too ;) As for your cookbooks, I'll look through them and ping you back on procedures to propose any of them for inclusion in upstream. All the best! -jay Tally ho, folks! -judd On Fri, Feb 24, 2012 at 9:59 AM, Matt Raym...@opscode.com wrote: There's a dashboard cookbook in the diablo/stable branch used by TryStack. https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks Thanks, Matt Ray Senior Technical Evangelist | Opscode Inc. m...@opscode.com | (512) 731-2218 Twitter, IRC, GitHub: mattray On Thu, Feb 23, 2012 at 2:10 PM, Mike Perezthin...@gmail.com wrote: Is anyone currently working on adding a Horizon cookbook? I would love to make this happen or work with someone who has already started on this. Thanks! -- Mike Perez On Monday, February 6, 2012 at 6:07 PM, Jay Pipes wrote: These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On Tue, Feb 28, 2012 at 2:44 PM, Jay Pipes jaypi...@gmail.com wrote: cc'ing list, since it's a great question and good follow-up conversation to have... On 02/28/2012 02:32 PM, andi abes wrote: Interesting. Would you mind doing a code review on Mary Newby's Swift All in One cookbook? Could sure use your experience :) https://review.openstack.org/#change,3613 Might be a good dip into the Gerrit world for ya, too ;) I think I might get to be a scratched record here, since the question was asked before is the goal of these cookbooks to support a SAIO env, or a more multi-noded version? a SIOA would be nice for devs and newbies, but hides some of the complexities. A multi-noded version would be more complex (and probably controversial around tradeoffs) but potentially have more value for users... (I obviously have my opinion, but curious as to where other folks are trying to drive this effort) I think that both are incredibly useful. With my as-yet-still-limited understanding of Chef, it would be possible to have both an SAIO and a multi-node Swift cookbook in the same repo, no? Or some combination of cookbooks and roles that would allow a node to install SAIO or a piece of the Swift multi-node puzzle? yes and no One neat feature of chef is it's search capability - being able to query the sever of where other pieces of the puzzle are located, which makes it very convenient for multi-node operations. E.g. for swift there are a few cookbooks floating around where by the rings are constructed by locating all the servers that are tagged as storage nodes (i.e. they have the appropriate role(s) assigned to them. While search is a neat capability, it does make the recipes more complex (recipes are part of cookbooks, that express the operations to be performed). So if the intent is to have the cookbooks serve as an newbie exemplar, showcasing openstack - its probably not a good idea. Other complexities arise when you start dealing with machine variably, that can be easily hidden in SAIO. Using swift as an example - the # and device names of disks. In SAIO, you just create a bunch of loopback devices... (at least the sample deployment docs do). On a more (dare I say) production environment, you'd want to discover what disks are available, and use the appropriate ones. That said - there could be recipes for both SIAO and multi-node. Users would then have to combine and apply the right set. But maybe that's not the full question... maybe a more complete question would be: is this effort geared towards producing deployments that can be considered production ready? -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
cc'ing Maru since his particular cookbook is being discussed here : On 02/28/2012 02:56 PM, andi abes wrote: yes and no One neat feature of chef is it's search capability - being able to query the sever of where other pieces of the puzzle are located, which makes it very convenient for multi-node operations. E.g. for swift there are a few cookbooks floating around where by the rings are constructed by locating all the servers that are tagged as storage nodes (i.e. they have the appropriate role(s) assigned to them. While search is a neat capability, it does make the recipes more complex (recipes are part of cookbooks, that express the operations to be performed). So if the intent is to have the cookbooks serve as an newbie exemplar, showcasing openstack - its probably not a good idea. Other complexities arise when you start dealing with machine variably, that can be easily hidden in SAIO. Using swift as an example - the # and device names of disks. In SAIO, you just create a bunch of loopback devices... (at least the sample deployment docs do). On a more (dare I say) production environment, you'd want to discover what disks are available, and use the appropriate ones. That said - there could be recipes for both SIAO and multi-node. Users would then have to combine and apply the right set. But maybe that's not the full question... maybe a more complete question would be: is this effort geared towards producing deployments that can be considered production ready? I believe most people would answer Yes. The openstack-chef upstream cookbooks should be geared towards production environments. I was just asking if there is the ability to have both production-ready recipes and try this out recipes all in the same repo. Sounds like that's not really a good thing, and if not, we should decide where the appropriate place to put try this out recipes should be? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Sorry I'm late to the game, but my swift chef recipes are on github as well. It's a fork of andi abes's recipes from back in mid 2011. They contain many an ugly hack and some half-baked ideas. BUT, if you want Ubuntu 10.04, munin, rsyslog, slogging and a bunch of other cool stuff, please give them a try. This is also a good impetus for me updating my swift repo. https://github.com/voxeldotnet/openstack-swift-chef Tally ho, folks! -judd On Fri, Feb 24, 2012 at 9:59 AM, Matt Ray m...@opscode.com wrote: There's a dashboard cookbook in the diablo/stable branch used by TryStack. https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks Thanks, Matt Ray Senior Technical Evangelist | Opscode Inc. m...@opscode.com | (512) 731-2218 Twitter, IRC, GitHub: mattray On Thu, Feb 23, 2012 at 2:10 PM, Mike Perez thin...@gmail.com wrote: Is anyone currently working on adding a Horizon cookbook? I would love to make this happen or work with someone who has already started on this. Thanks! -- Mike Perez On Monday, February 6, 2012 at 6:07 PM, Jay Pipes wrote: These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Judd Maltin T: 917-882-1270 F: 501-694-7809 A loving heart is never wrong. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Is anyone currently working on adding a Horizon cookbook? I would love to make this happen or work with someone who has already started on this. Thanks! -- Mike Perez On Monday, February 6, 2012 at 6:07 PM, Jay Pipes wrote: These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
There's a dashboard cookbook in the diablo/stable branch used by TryStack. https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks Thanks, Matt Ray Senior Technical Evangelist | Opscode Inc. m...@opscode.com | (512) 731-2218 Twitter, IRC, GitHub: mattray On Thu, Feb 23, 2012 at 2:10 PM, Mike Perez thin...@gmail.com wrote: Is anyone currently working on adding a Horizon cookbook? I would love to make this happen or work with someone who has already started on this. Thanks! -- Mike Perez On Monday, February 6, 2012 at 6:07 PM, Jay Pipes wrote: These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
We're workin' on it... Had some issues with AppArmor crashing a node this morning... https://github.com/trystack/openstack-chef/tree/stable/diablo -jay On 02/16/2012 10:25 AM, andi abes wrote: On Wed, Feb 8, 2012 at 11:42 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 02/08/2012 01:40 AM, Monty Taylor wrote: On 02/07/2012 08:32 PM, Jay Pipes wrote: Well, in my original email I proposed using the NTT PF Lab branch point for the stable/diablo branch of the upstream chef repos. If we can get a casual consensus from folks that this is OK, I will go ahead and push that to Gerrit. Please +1 if you are cool with that. This will allow us to have a branch of the upstream cookbooks that aligns with the core projects. Ping me when you want to do that... jeblair and I can handle getting the branch in once you have it. I'd like to do it today, please. Find me on IRC and we'll get this done. Best, -jay Did this branch landed somewhere? As a side note - seems that the NTT cookbooks don't contain swift support. The swift cookbook in crowbar function standalone, and has a relatively heavily annotated default attribute file as a mini howto guide. _ Mailing list: https://launchpad.net/~__openstack https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~__openstack https://launchpad.net/~openstack More help : https://help.launchpad.net/__ListHelp https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On Wed, Feb 8, 2012 at 11:42 AM, Jay Pipes jaypi...@gmail.com wrote: On 02/08/2012 01:40 AM, Monty Taylor wrote: On 02/07/2012 08:32 PM, Jay Pipes wrote: Well, in my original email I proposed using the NTT PF Lab branch point for the stable/diablo branch of the upstream chef repos. If we can get a casual consensus from folks that this is OK, I will go ahead and push that to Gerrit. Please +1 if you are cool with that. This will allow us to have a branch of the upstream cookbooks that aligns with the core projects. Ping me when you want to do that... jeblair and I can handle getting the branch in once you have it. I'd like to do it today, please. Find me on IRC and we'll get this done. Best, -jay Did this branch landed somewhere? As a side note - seems that the NTT cookbooks don't contain swift support. The swift cookbook in crowbar function standalone, and has a relatively heavily annotated default attribute file as a mini howto guide. __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/~openstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/08/2012 01:40 AM, Monty Taylor wrote: On 02/07/2012 08:32 PM, Jay Pipes wrote: Well, in my original email I proposed using the NTT PF Lab branch point for the stable/diablo branch of the upstream chef repos. If we can get a casual consensus from folks that this is OK, I will go ahead and push that to Gerrit. Please +1 if you are cool with that. This will allow us to have a branch of the upstream cookbooks that aligns with the core projects. Ping me when you want to do that... jeblair and I can handle getting the branch in once you have it. I'd like to do it today, please. Find me on IRC and we'll get this done. Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Hi Jay, Thanks for taking the initiative to send this out! I added comments to your points are inline below: Proposal for Alignment == I think the following steps would be good to get done by the time Essex rolls out the door in April: 1) Create a stable/diablo branch of the openstack/openstack-chef cookbook repo and maintain it in the same way that we maintain stable branches for core OpenStack projects. I propose we use the branch point that NTT PF Lab used to create their fork of the upstream repo. I like the idea of maintaining a stable set of cookbooks for the official releases. The NTT branch sounds fine to me as a starting point for diablo. At the time of the diablo release I was maintaining cookbooks for SmokeStack here: https://github.com/dprince/openstack_cookbooks. Using these cookbooks around the date of the diablo release would be an option as well. 2) Work with Matt Ray and other Chef experts to combine any and all best practices that may be contained in the non-official cookbook repos into the upstream official repository. From a cursory overview, there are some differences in how databags are handled, how certs are handled, how certain cookbooks are constructed, and of course differences in the actual cookbooks in the repos themselves. 3) Consolidate documentation on how to use the cookbooks, the best practices used in constructing the cookbooks, and possibly some videos/tutorials walking folks through this critical piece of the OpenStack puzzle. This sounds great. 4) Create Jenkins builders for stable branch deployment testing. We currently test the official development cookbooks by way of SmokeStack gates on all core OpenStack projects. Would be great to get the same testing automated for non-development branches of the cookbooks. SmokeStack would easily support testing stable releases. In fact it be a lot easier to pull off stable release testing than it is to chase trunk like I'm currently doing :) I actually have a 'Libvirt Mysql Milestone Proposed (Diablo)' configuration in SmokeStack. I just haven't been running it mostly because I was focused on upstream releases and commits. Limited resources and time Getting more people involved would be great. Thoughts and criticism most welcome, and apologies in advance if I got any of the above history wrong. Feel free to correct me! Best, -jay One final note: We are looking at adding dual support for Fedora/puppet and Ubuntu/chef to SmokeStack in the near future. A guy named Derek Higgins from Red Hat has made excellent progress on this front. -- Dan Prince princ...@alumni.jmu.edu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
apologies for possible duplicates - some replies last night were from the wrong email account (and didn't make it to the list) On Mon, Feb 6, 2012 at 9:37 PM, Jesse Andrews anotherje...@gmail.comwrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Right. The results of those efforts are here: https://github.com/dellcloudedge/crowbar/tree/openstack-os-build/barclamps (these are git submodules). There are barclamps for swift,keystone, nova and horizon. In each barclamp you'll find a chef directory with sub-directories for cookbooks and databags. Most are designed to work both within and outside of crowbar (by using default attributes to replace values otherwise set by crowbar). Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.orgjust like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki:
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Jay and All, I have a couple of questions about the goals of the proposal. 1. What is the origin for the package declarations? - PPA's - Tarball - OS Packages (e.g. whatever happens to be in ubuntu's 11.04 or 11.10 repo?) - Git checkout 2. Are these cookbooks meant to be run from a chef-server or as a chef-solo run? 3. In the case of stable branches, are the cookbooks going to be targeted at a specific release point inside of a branch? Or will they always be updated to follow the head of a branch. 4. Are the cookbooks going to be aimed at all first class hypervisors? As a maintainer of yet another repo of chef recipes (RCB Deploy) and committer to Crowbar, I am happy to collaborate on the consolidation. --shep On Feb 7, 2012, at 12:21 AM, Monty Taylor wrote: On 02/06/2012 06:07 PM, Jay Pipes wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) I would like to get these to have a stable/diablo branch. These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. I think Dan and Brian are also deploying kvm as part of smokestack. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... If these are a fork of openstack/openstack-chef, could we perhaps make these the basis of a stable/diablo branch in openstack-chef? ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Not sure on #1 and #3, but as for #2 and #4: #2) Most of the openstack/openstack-chef cookbooks should be able to run with chef-server or chef-solo. They are mostly tested with chef-server however. #4) I don't see why not, I'd love to support a variety of different configurations/options. Right now the openstack/openstack-chef project is used for Libvirt and Xenserver deployments using some MySQL/Postgres/SQLite options and Keystone/No-keystone auth. Loves top-posting, Lamar -Original Message- From: Justin Shepherd jshep...@rackspace.com Sent: Tuesday, February 7, 2012 11:56am To: Jay Pipes jaypi...@gmail.com Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net Subject: Re: [Openstack] [CHEF] Aligning Cookbook Efforts Jay and All, I have a couple of questions about the goals of the proposal. 1. What is the origin for the package declarations? - PPA's - Tarball - OS Packages (e.g. whatever happens to be in ubuntu's 11.04 or 11.10 repo?) - Git checkout 2. Are these cookbooks meant to be run from a chef-server or as a chef-solo run? 3. In the case of stable branches, are the cookbooks going to be targeted at a specific release point inside of a branch? Or will they always be updated to follow the head of a branch. 4. Are the cookbooks going to be aimed at all first class hypervisors? As a maintainer of yet another repo of chef recipes (RCB Deploy) and committer to Crowbar, I am happy to collaborate on the consolidation. --shep On Feb 7, 2012, at 12:21 AM, Monty Taylor wrote: On 02/06/2012 06:07 PM, Jay Pipes wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) I would like to get these to have a stable/diablo branch. These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. I think Dan and Brian are also deploying kvm as part of smokestack. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... If these are a fork of openstack/openstack-chef, could we perhaps make these the basis of a stable/diablo branch in openstack-chef? ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
I think Jay did a good job outlining the lineages and scope of the assorted cookbook efforts so far. My Anso-based fork at github.com/mattray/openstack-cookbooks was the basis for a few public and private forks and strictly focused on multi-node deployments of stable releases. A lot of this went into Dell's efforts with Crowbar and they've continued to develop for Diablo and future releases while working with Rackspace Cloud Builders. While I've been working with them primarily since the Diablo release, I've also done work with folks who do not want to use Crowbar, and pulling those cookbooks out of the barclamps hasn't always been straightforward. After discussing the state of Chef cookbooks out there with Jay and folks at Dell, I plan on forking off of github.com/openstack/openstack-chef/tree/stable/diablo once it's there and getting it synced up with the efforts in the Crowbar cookbooks. I'll be pushing into Opscode's repo and hopefully sending patches upstream to both efforts. My repo will probably not be a good candidate for an upstream provider since updates may be sporadic since I don't only work on OpenStack. I won't follow trunk and I'll continue to strictly focus on deploying the current stable release. It's likely at some point my repo will pull in support for multiple implementations of the Swift API (Ceph, Gluster, etc.) and hopefully additional hypervisors and databases and RHEL support; it's driven by whoever I'm currently working with. I also plan on unforking the many non-OpenStack cookbooks from the repo and pushing any changes upstream to help keep things simple. While the Cactus documentation I wrote is stale (http://bit.ly/OSChef), I'll update it going forward and try to keep my repo well-documented as far as what all the pieces are and usage. On Jay's suggestion I'll keep the content synced between the Opscode wiki and the README.md. My current #1 priority is getting knife-openstack fixed, then I'll get working on this again. Thanks, Matt Ray Senior Technical Evangelist | Opscode Inc. m...@opscode.com | (512) 731-2218 Twitter, IRC, GitHub: mattray On Mon, Feb 6, 2012 at 8:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Thanks for the update, Matt. Comments inline... On 02/07/2012 10:16 PM, Matt Ray wrote: I think Jay did a good job outlining the lineages and scope of the assorted cookbook efforts so far. My Anso-based fork at github.com/mattray/openstack-cookbooks was the basis for a few public and private forks and strictly focused on multi-node deployments of stable releases. A lot of this went into Dell's efforts with Crowbar and they've continued to develop for Diablo and future releases while working with Rackspace Cloud Builders. While I've been working with them primarily since the Diablo release, I've also done work with folks who do not want to use Crowbar, and pulling those cookbooks out of the barclamps hasn't always been straightforward. From what I gather, the reason it's not always straightforward is because the cookbooks in the Crowbar barclamps are quite opinionated and make a number of assumptions about the underlying hardware or environment. I will begin researching how to pull work that's been done in the Crowbar barclamp cookbooks into the upstream chef repo -- and in the process come up with some documentation on how to create the cookbooks in a way that is flexible enough to service multiple environments by using roles, attributes and databags. Chef is still new to me, though, so I will likely lean on you heavily for advice on best practices. I'll put the documentation directly into the upstream Chef repos unless folks think the wiki or some other place on openstack.org would be a better choice? After discussing the state of Chef cookbooks out there with Jay and folks at Dell, I plan on forking off of github.com/openstack/openstack-chef/tree/stable/diablo once it's there and getting it synced up with the efforts in the Crowbar cookbooks. I'll be pushing into Opscode's repo and hopefully sending patches upstream to both efforts. Well, in my original email I proposed using the NTT PF Lab branch point for the stable/diablo branch of the upstream chef repos. If we can get a casual consensus from folks that this is OK, I will go ahead and push that to Gerrit. Please +1 if you are cool with that. This will allow us to have a branch of the upstream cookbooks that aligns with the core projects. My repo will probably not be a good candidate for an upstream provider since updates may be sporadic since I don't only work on OpenStack. I won't follow trunk and I'll continue to strictly focus on deploying the current stable release. Understood. There are plenty of others who are interested in following trunk and ensuring that additions to trunk make their way into the stable branches. Generally, the way that OpenStack projects work is that changes are first proposed to the development trunk and then a separate set of stable maintainers will cherry-pick relevant hotfixes into the stable branch they are maintaining -- resolving any merge conflicts that may arise during that operation. I think all that would be needed from you to keep the deployment wheels greased would be a short email notification to the mailing list when you add or change something substantial in your downstream repos that you feel would benefit the broader community. Then the maintainers of the upstream stable cookbooks repo can simply check your repo out, determine if and how those commits can be applied to the development branch and work with the development branch contributors to get the aforementioned development - stable cherry-pick process going. It's likely at some point my repo will pull in support for multiple implementations of the Swift API (Ceph, Gluster, etc.) and hopefully additional hypervisors and databases and RHEL support; it's driven by whoever I'm currently working with. I also plan on unforking the many non-OpenStack cookbooks from the repo and pushing any changes upstream to help keep things simple. Excellent. I think this is definitely something that the broader community would be eager to use and contribute to. While the Cactus documentation I wrote is stale (http://bit.ly/OSChef), I'll update it going forward and try to keep my repo well-documented as far as what all the pieces are and usage. On Jay's suggestion I'll keep the content synced between the Opscode wiki and the README.md. My current #1 priority is getting knife-openstack fixed, then I'll get working on this again. Rock on. Thanks Matt! -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/06/2012 11:53 PM, Vishvananda Ishaya wrote: On Feb 6, 2012, at 6:37 PM, Jesse Andrews wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. They were the basis of dan and matt's cookbooks, but they are now ancient history. i've been using them as a repository for a few helper devstack recipes, but waldon pulled those out into a separate repo so it is fine if we torch them. Sounds good. Feel free to blast them away... -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/07/2012 08:32 PM, Jay Pipes wrote: Thanks for the update, Matt. Comments inline... On 02/07/2012 10:16 PM, Matt Ray wrote: I think Jay did a good job outlining the lineages and scope of the assorted cookbook efforts so far. My Anso-based fork at github.com/mattray/openstack-cookbooks was the basis for a few public and private forks and strictly focused on multi-node deployments of stable releases. A lot of this went into Dell's efforts with Crowbar and they've continued to develop for Diablo and future releases while working with Rackspace Cloud Builders. While I've been working with them primarily since the Diablo release, I've also done work with folks who do not want to use Crowbar, and pulling those cookbooks out of the barclamps hasn't always been straightforward. From what I gather, the reason it's not always straightforward is because the cookbooks in the Crowbar barclamps are quite opinionated and make a number of assumptions about the underlying hardware or environment. I will begin researching how to pull work that's been done in the Crowbar barclamp cookbooks into the upstream chef repo -- and in the process come up with some documentation on how to create the cookbooks in a way that is flexible enough to service multiple environments by using roles, attributes and databags. Chef is still new to me, though, so I will likely lean on you heavily for advice on best practices. I'll put the documentation directly into the upstream Chef repos unless folks think the wiki or some other place on openstack.org would be a better choice? After discussing the state of Chef cookbooks out there with Jay and folks at Dell, I plan on forking off of github.com/openstack/openstack-chef/tree/stable/diablo once it's there and getting it synced up with the efforts in the Crowbar cookbooks. I'll be pushing into Opscode's repo and hopefully sending patches upstream to both efforts. Well, in my original email I proposed using the NTT PF Lab branch point for the stable/diablo branch of the upstream chef repos. If we can get a casual consensus from folks that this is OK, I will go ahead and push that to Gerrit. Please +1 if you are cool with that. This will allow us to have a branch of the upstream cookbooks that aligns with the core projects. Ping me when you want to do that... jeblair and I can handle getting the branch in once you have it. My repo will probably not be a good candidate for an upstream provider since updates may be sporadic since I don't only work on OpenStack. I won't follow trunk and I'll continue to strictly focus on deploying the current stable release. Understood. There are plenty of others who are interested in following trunk and ensuring that additions to trunk make their way into the stable branches. Generally, the way that OpenStack projects work is that changes are first proposed to the development trunk and then a separate set of stable maintainers will cherry-pick relevant hotfixes into the stable branch they are maintaining -- resolving any merge conflicts that may arise during that operation. I think all that would be needed from you to keep the deployment wheels greased would be a short email notification to the mailing list when you add or change something substantial in your downstream repos that you feel would benefit the broader community. Then the maintainers of the upstream stable cookbooks repo can simply check your repo out, determine if and how those commits can be applied to the development branch and work with the development branch contributors to get the aforementioned development - stable cherry-pick process going. It's likely at some point my repo will pull in support for multiple implementations of the Swift API (Ceph, Gluster, etc.) and hopefully additional hypervisors and databases and RHEL support; it's driven by whoever I'm currently working with. I also plan on unforking the many non-OpenStack cookbooks from the repo and pushing any changes upstream to help keep things simple. Excellent. I think this is definitely something that the broader community would be eager to use and contribute to. While the Cactus documentation I wrote is stale (http://bit.ly/OSChef), I'll update it going forward and try to keep my repo well-documented as far as what all the pieces are and usage. On Jay's suggestion I'll keep the content synced between the Opscode wiki and the README.md. My current #1 priority is getting knife-openstack fixed, then I'll get working on this again. Rock on. Thanks Matt! -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [CHEF] Aligning Cookbook Efforts
Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the fact that folks are just trying to stand up complex clouds and Get Things Done, and documentation is annoying to write ;) Proposal for Alignment == I think the following steps would be good to get done by the time Essex rolls out the door in April: 1) Create a stable/diablo branch of the openstack/openstack-chef cookbook repo and maintain it in the same way that we maintain stable branches for core OpenStack projects. I propose we use the branch
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Awesome leadership on this Jay! (And Matt) You should come down to Austin more often :) Definitely seems like an area with a lot of duplicated effort. Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the fact that folks are just trying to stand up complex clouds and Get Things Done, and documentation is annoying to write ;) Proposal for Alignment == I think the following steps would be good to get done by the time Essex rolls out the door in April: 1) Create a stable/diablo branch of the openstack/openstack-chef
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the fact that folks are just trying to stand up complex clouds and Get Things Done, and documentation is annoying to write ;) Proposal for Alignment
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/06/2012 09:37 PM, Jesse Andrews wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. OK. Are you in agreement about the proposal in my email? Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. Yup, no prob. Best, -jay Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipesjaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the fact that folks are just trying to stand up complex clouds and Get Things Done, and documentation is annoying to
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
RCB deploy has a set of chef cookbooks we use for diablo all in one testing (minus swift) in addition to the work we have been doing with the crowbar team. Since we seem to be adding to the problem, we'd be happy to collaborate on the consolidation.. Jason On Feb 6, 2012, at 20:56, Jesse Andrews anotherje...@gmail.com wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On Feb 6, 2012, at 6:37 PM, Jesse Andrews wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. They were the basis of dan and matt's cookbooks, but they are now ancient history. i've been using them as a repository for a few helper devstack recipes, but waldon pulled those out into a separate repo so it is fine if we torch them. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
Jesse, right. The results are here: https://github.com/dellcloudedge/crowbar/tree/openstack-os-build/barclamps . With separate repos for nova, swift, keystone and horizon ( at this location, they're git submodules) The v1.2 tag deploys diablo/stable. Most Cookbooks are written to be useable within and without crowbar, thoug we mostly test within crowbar. (writing from home, hence the gmail, rather than @dell address) On Feb 6, 2012, at 9:37 PM, Jesse Andrews anotherje...@gmail.com wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
I've submitted a Swift AIO cookbook for review: https://review.openstack.org/#change,3613 It follows the latest single-node AIO instructions pretty much to the letter, so the resulting environment is well-documented. We use this cookbook as the basis for building Swift development environments here at Internap. Thanks, Maru On 2012-02-06, at 6:07 PM, Jay Pipes wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
The alignment proposal sounds great, and would definitely help reduce redundancy. However, it might be useful to define clear goals of the resulting deployment using these cookbooks. As an example - Looking at the anso recipes for swift - they appear to deploy a SAIO swift cluster. The Crowbar cookbook assumes a multi node deployment. Similarly for nova - the official cookbooks appear to focus only on flat networking (unless I'm missing something) while the Crowbar version supports multiple network configs ( e.g. Vlan). OTH, the official recipes support both MySQL and Postgres, while crowbar only supports MySQL. ( the above not intended to recommend brands of sliced bread ;) The above raises a few questions ( and I'm sure there might be more): - SAIO or multi node ? - possibly repeat of the above - are the cookbooks to be used beyond unit testing, or just serve as an example? - what coverage for the breadth of options ? Or stated differently - are the cookbooks prescriptive and opinionated about deployments, or flexible? - does the above apply just to openstack components, or 3rd party dependencies ? A. On Feb 6, 2012, at 11:53 PM, Vishvananda Ishaya vishvana...@gmail.com wrote: On Feb 6, 2012, at 6:37 PM, Jesse Andrews wrote: I know that the RCB deploy team works with the Crowbar team on chef recipes for that project. Regarding the github.com/ansolabs github.com/rcb recipes - I'll have to delegate to Vishy who worked on those. They were the basis of dan and matt's cookbooks, but they are now ancient history. i've been using them as a repository for a few helper devstack recipes, but waldon pulled those out into a separate repo so it is fine if we torch them. Jesse On Mon, Feb 6, 2012 at 6:07 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says*
Re: [Openstack] [CHEF] Aligning Cookbook Efforts
On 02/06/2012 06:07 PM, Jay Pipes wrote: Hi Stackers, tl;dr - There are myriad Chef cookbooks out there in the ecosystem and locked up behind various company firewalls. It would be awesome if we could agree to: * Align to a single origin repository for OpenStack cookbooks * Consolidate OpenStack Chef-based deployment experience into a single knowledge base * Have branches on the origin OpenStack cookbooks repository that align with core OpenStack projects * Automate the validation and testing of these cookbooks on multiple supported versions of the OpenStack code base Details --- Current State of Forks == Matt Ray and I tried to outline the current state of the various OpenStack Chef cookbooks this past Thursday, and we came up with the following state of affairs: ** The official OpenStack Chef cookbooks ** https://github.com/openstack/openstack-chef These chef cookbooks are the ones maintained mostly by Dan Prince and Brian Lamar and these are the cookbooks used by the SmokeStack project. The cookbooks contained in the above repo can install all the core OpenStack projects with the exception of Swift and Horizon. This repo is controlled by the Gerrit instance at review.openstack.org just like other core OpenStack projects. However, these cookbooks DO NOT currently have a stable/diablo branch -- they are updated when the development trunks of any OpenStack project merges a commit that requires deployment or configuration-related changes to their associated cookbook. Important note: it's easy for Dan and Brian to know when updates to these cookbooks are necessary -- SmokeStack will bomb out if a deployment-affecting configuration change hits a core project trunk :) I would like to get these to have a stable/diablo branch. These cookbooks are the ONLY cookbooks that contain stuff for deploying with XenServer, AFAICT. I think Dan and Brian are also deploying kvm as part of smokestack. ** NTT PF Lab Diablo Chef cookbooks ** https://github.com/ntt-pf-lab/openstack-chef/ So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, because they needed a set of Chef cookbooks for OpenStack that functioned for the Diablo code base. While Nov 11, 2011, is not the *exact* date of the Diablo release, these cookbooks do in fact work for a Diablo install -- Nati Ueno is using them for the FreeCloud deployment so we know they work... If these are a fork of openstack/openstack-chef, could we perhaps make these the basis of a stable/diablo branch in openstack-chef? ** OpsCode OpenStack Chef Cookbooks ** Matt Ray from OpsCode created a set of cookbooks for OpenStack for the Cactus release of OpenStack: https://github.com/mattray/openstack-cookbooks http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef These cookbooks were forked from the Anso Labs' original OpenStack cookbooks from the Bexar release and were the basis for the Chef work that Dell did for Crowbar. Crowbar was originally based on Cactus, and according to Matt, the repositories of OpenStack cookbooks that OpsCode houses internally and uses most often are Cactus-based cookbooks. (Matt, please correct me if I am wrong here...) ** Rackspace CloudBuilders OpenStack Chef Cookbooks ** The RCB team also has a repository of OpenStack Chef cookbooks: https://github.com/cloudbuilders/openstack-cookbooks Now, GitHub *says* that these cookbooks were forked from the official upstream cookbooks, but I do not think that is correct. Looking at this repo, I believe that this repo was *actually* forked from the Anso Labs OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical. ** Anso Labs OpenStack Chef Cookbooks ** These older cookbooks are in this repo: https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks Interestingly, this repo DOES contain a cookbook for Swift. Current State of Documentation == Documentation for best practices on using Chef for your OpenStack deployments is, well, a bit scattered. Matt Ray has some good information on the README on his cookbook repo and the OpsCode wiki: https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef But it is unfortunately not going to help people looking to deploy Diablo and later versions of OpenStack. Most of the other repos contain virtually no documentation on using the cookbooks or how they are written. I have a suspicion that one of the reasons that there has been such a proliferation of cookbooks has been the lack of documentation pointing people to an appropriate repo, how to use the cookbooks properly, and what the best practices for deployment are. That, and the fact that folks are just trying to stand up complex clouds and Get Things Done, and documentation is annoying to
