Re: [OpenStack-Infra] Development environments for infra's puppet modules
My tool of choice is vagrant - it's super easy to create a Vagrantfile, write a quick shell script that idempotently installs any required puppet modules, and then run puppet itself. An older example is the puppet-storyboard module: git checkout, vagrant up, and you're off to the races. https://github.com/openstack-infra/puppet-storyboard/blob/master/Vagrantfile That approach can be extended to simulating any openstack infra node including slaves, servers, etc https://krotscheck.net/2016/06/01/how-to-simulate-an-openstack-infra-slave.html Michael ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
(sorry, mail client failure) On Tue, Aug 30, 2016 at 4:59 PM, Joshua Hesketh wrote: > > > On Fri, Aug 26, 2016 at 1:31 AM, Simon McCartney > wrote: >> >> >> >> > However, I'm not sure Vagrant provides a good solution for testing puppet >> modules in isolation (I think it's great for the >> system-config/project-config scenario, where you want to see how applying >> the full set of required puppet modules on to an empty VM provides a >> working system), it's harder to test standing up zuul without also setting >> up a few other components, so puppet-zuul (for example) may not take >> advantage of Vagrant directly, but may benefit from beaker[1] or >> test-kitchen[2] work (I think that conversation has happened before but I >> wasn't directly involved at the time) >> >> > So I think this is the interesting part for vagrant. The infra bootstrapping will install other components (as mentioned) which is useful when you want to develop on something intended for OpenStack's infra. However if we want the puppet modules to be more generic and used outside of infra's use case, is this sufficient or should we look towards something more generic (such as vagrant)? I'm not really sure how this would look as modules may have complicated inter-dependencies or assumptions about the environment/machine. How do other puppet developers do their modules upstream? Cheers, Josh ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
On Fri, Aug 26, 2016 at 1:31 AM, Simon McCartney wrote: > > > > However, I'm not sure Vagrant provides a good solution for testing puppet > modules in isolation (I think it's great for the > system-config/project-config scenario, where you want to see how applying > the full set of required puppet modules on to an empty VM provides a > working system), it's harder to test standing up zuul without also setting > up a few other components, so puppet-zuul (for example) may not take > advantage of Vagrant directly, but may benefit from beaker[1] or > test-kitchen[2] work (I think that conversation has happened before but I > wasn't directly involved at the time) > > [1] https://github.com/puppetlabs/beaker > [2] https://github.com/neillturner/kitchen-puppet & http://kitchen.ci/ > > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
On Thu, Aug 25, 2016 at 7:48 AM, James E. Blair wrote: > Joshua Hesketh writes: > >> There's probably further discussions here but I don't have enough knowledge >> in this area to comment further. The aim though should be to make it easy >> to bootstrap a server with the module you're developing on so you can >> easily verify and debug your changes. > > Don't forget the documentation we already have on this subject: > > http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet This is what I've been using for years. Load up a new instance in a cloud or locally on KVM, follow the documentation and test the change I'm writing/reviewing. It's not all-encompassing (multi-node interactions are still time-consuming to test properly), and it's not fast, but it has been effective for most changes. Plus, it doesn't require maintaining anything, those docs have stayed the same and accurate for years. That said, I think your idea has merit. Making testing changes faster and easier is always helpful, as long someone is willing to do the work to maintain it (and it doesn't turn into our long defunct "Running your own CI infrastructure" docs). From where I'm sitting Vagrant seems as good a way as any. -- Elizabeth Krumbach Joseph || Lyz || pleia2 ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
> 2) Is Vagrant a good fit for this? Otherwise should we consider an > ansible-playbook to bootstrap an environment? > Just while you mention ansible, we've been looking at the infra-ansible[1] & ansible-role-cloud-launcher[2] that Ricardo and others have been working on, and as that project is relatively early in it's lifecycle we should work on getting a friendly, safe, low-friction environment for working on that suite, the current model involved a lot of manual copying stuff around or using private repos for development, I'm sure we can find a better way than that [1] https://github.com/openstack-infra/infra-ansible [2] https://github.com/openstack/ansible-role-cloud-launcher ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
> > We should look for a way to make developing, debugging and testing our > puppet modules locally easier and more consistent. > Short of bootstrapping an entire clone of openstack-infra, how do > This is pretty much what we do at the moment to maintain our internal openstack-infra CI pipeline, we're still running a puppet master with a single control repo (i.e. we're not split into system-config & project-config yet) We have a Vagrantfile that allows us to standup a puppet master with our config & openstack-infra/config & hieradata mounted from local checkouts on the workstation, this allows us to tweak our private modules directly, import external modules via modules.env & tweak site.pp directly we then have further guest vm definitions in our Vagrantfile to match the host/role that's being worked on (as we're using site.pp, hostnames need to match entries in site.pp, so we use vagrant-hostmanager to maintain /etc/hosts across the guest VMs so that when we stand up a guest vm, the puppet master compiles a catalogue that closely matches production. > developers currently set up an environment to investigate how a puppet > module behaves? This brings me to my first question: > 1) Do we want to find/provide a way to set up a consistent development > environment? > Vagrant could be a useful way of providing a consistent development > environment for those working on infra's puppet modules. This comes up in > light of https://review.openstack.org/#/c/355273 which was split out from > a larger change due to debate over any vagrant precedent. This change was > in turn based upon this documented example of simulating OpenStack Infra > environments for testing: https://krotscheck.net/2016/06 > /01/how-to-simulate-an-openstack-infra-slave.html > Currently the only module (that a quick grep found for me) providing a > vagrant file is puppet-storyboard. > 2) Is Vagrant a good fit for this? Otherwise should we consider an > ansible-playbook to bootstrap an environment? > I think Vagrant is very useful for this, it provides a decent abstraction over the various virtualisation systems you may wish to use (virtualbox, kvm, docker - we're only using virtualbox internally as it's the most widely supported provider in Vagrant) - setting up the required networking, mapping filesystems into VMs & managing /etc/hosts on guests & on the host as required. However, I'm not sure Vagrant provides a good solution for testing puppet modules in isolation (I think it's great for the system-config/project-config scenario, where you want to see how applying the full set of required puppet modules on to an empty VM provides a working system), it's harder to test standing up zuul without also setting up a few other components, so puppet-zuul (for example) may not take advantage of Vagrant directly, but may benefit from beaker[1] or test-kitchen[2] work (I think that conversation has happened before but I wasn't directly involved at the time) [1] https://github.com/puppetlabs/beaker [2] https://github.com/neillturner/kitchen-puppet & http://kitchen.ci/ ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Development environments for infra's puppet modules
Joshua Hesketh writes: > There's probably further discussions here but I don't have enough knowledge > in this area to comment further. The aim though should be to make it easy > to bootstrap a server with the module you're developing on so you can > easily verify and debug your changes. Don't forget the documentation we already have on this subject: http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra