Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Michael Krotscheck]

My tool of choice is vagrant - it's super easy to create a Vagrantfile,
write a quick shell script that idempotently installs any required puppet
modules, and then run puppet itself. An older example is the
puppet-storyboard module: git checkout, vagrant up, and you're off to the
races.

https://github.com/openstack-infra/puppet-storyboard/blob/master/Vagrantfile

That approach can be extended to simulating any openstack infra node
including slaves, servers, etc
https://krotscheck.net/2016/06/01/how-to-simulate-an-openstack-infra-slave.html

Michael
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Joshua Hesketh]

(sorry, mail client failure)

On Tue, Aug 30, 2016 at 4:59 PM, Joshua Hesketh 
wrote:

>
>
> On Fri, Aug 26, 2016 at 1:31 AM, Simon McCartney 
> wrote:
>>
>>
>> 
>>
> However, I'm not sure Vagrant provides a good solution for testing puppet
>> modules in isolation (I think it's great for the
>> system-config/project-config scenario, where you want to see how applying
>> the full set of required puppet modules on to an empty VM provides a
>> working system), it's harder to test standing up zuul without also setting
>> up a few other components, so puppet-zuul (for example) may not take
>> advantage of Vagrant directly, but may benefit from beaker[1] or
>> test-kitchen[2] work (I think that conversation has happened before but I
>> wasn't directly involved at the time)
>>
>>
>
 So I think this is the interesting part for vagrant. The infra
bootstrapping will install other components (as mentioned) which is useful
when you want to develop on something intended for OpenStack's infra.
However if we want the puppet modules to be more generic and used outside
of infra's use case, is this sufficient or should we look towards something
more generic (such as vagrant)?

I'm not really sure how this would look as modules may have complicated
inter-dependencies or assumptions about the environment/machine. How do
other puppet developers do their modules upstream?

Cheers,
Josh
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Joshua Hesketh]

On Fri, Aug 26, 2016 at 1:31 AM, Simon McCartney  wrote:
>
>
> 
>
However, I'm not sure Vagrant provides a good solution for testing puppet
> modules in isolation (I think it's great for the
> system-config/project-config scenario, where you want to see how applying
> the full set of required puppet modules on to an empty VM provides a
> working system), it's harder to test standing up zuul without also setting
> up a few other components, so puppet-zuul (for example) may not take
> advantage of Vagrant directly, but may benefit from beaker[1] or
> test-kitchen[2] work (I think that conversation has happened before but I
> wasn't directly involved at the time)
>
> [1] https://github.com/puppetlabs/beaker
> [2] https://github.com/neillturner/kitchen-puppet & http://kitchen.ci/
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Elizabeth K. Joseph]

On Thu, Aug 25, 2016 at 7:48 AM, James E. Blair  wrote:
> Joshua Hesketh  writes:
>
>> There's probably further discussions here but I don't have enough knowledge
>> in this area to comment further. The aim though should be to make it easy
>> to bootstrap a server with the module you're developing on so you can
>> easily verify and debug your changes.
>
> Don't forget the documentation we already have on this subject:
>
> http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet

This is what I've been using for years. Load up a new instance in a
cloud or locally on KVM, follow the documentation and test the change
I'm writing/reviewing. It's not all-encompassing (multi-node
interactions are still time-consuming to test properly), and it's not
fast, but it has been effective for most changes. Plus, it doesn't
require maintaining anything, those docs have stayed the same and
accurate for years.

That said, I think your idea has merit. Making testing changes faster
and easier is always helpful, as long someone is willing to do the
work to maintain it (and it doesn't turn into our long defunct
"Running your own CI infrastructure" docs). From where I'm sitting
Vagrant seems as good a way as any.

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Simon McCartney]

> 2) Is Vagrant a good fit for this? Otherwise should we consider an
> ansible-playbook to bootstrap an environment?
>

Just while you mention ansible, we've been looking at the infra-ansible[1]
& ansible-role-cloud-launcher[2] that Ricardo and others have been working
on, and as that project is relatively early in it's lifecycle we should
work on getting a friendly, safe, low-friction environment for working on
that suite, the current model involved a lot of manual copying stuff around
or using private repos for development, I'm sure we can find a better way
than that

[1] https://github.com/openstack-infra/infra-ansible
[2] https://github.com/openstack/ansible-role-cloud-launcher
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[Simon McCartney]

>
> We should look for a way to make developing, debugging and testing our
> puppet modules locally easier and more consistent.
> Short of bootstrapping an entire clone of openstack-infra, how do
>

This is pretty much what we do at the moment to maintain our internal
openstack-infra CI pipeline, we're still running a puppet master with a
single control repo (i.e. we're not split into system-config &
project-config yet)

We have a Vagrantfile that allows us to standup a puppet master with our
config & openstack-infra/config & hieradata mounted from local checkouts on
the workstation, this allows us to tweak our private modules directly,
import external modules via modules.env & tweak site.pp directly

we then have further guest vm definitions in our Vagrantfile to match the
host/role that's being worked on (as we're using site.pp, hostnames need to
match entries in site.pp, so we use vagrant-hostmanager to maintain
/etc/hosts across the guest VMs so that when we stand up a guest vm, the
puppet master compiles a catalogue that closely matches production.


> developers currently set up an environment to investigate how a puppet
> module behaves? This brings me to my first question:
> 1) Do we want to find/provide a way to set up a consistent development
> environment?
> Vagrant could be a useful way of providing a consistent development
> environment for those working on infra's puppet modules. This comes up in
> light of https://review.openstack.org/#/c/355273 which was split out from
> a larger change due to debate over any vagrant precedent. This change was
> in turn based upon this documented example of simulating OpenStack Infra
> environments for testing: https://krotscheck.net/2016/06
> /01/how-to-simulate-an-openstack-infra-slave.html
> Currently the only module (that a quick grep found for me) providing a
> vagrant file is puppet-storyboard.
> 2) Is Vagrant a good fit for this? Otherwise should we consider an
> ansible-playbook to bootstrap an environment?
>

I think Vagrant is very useful for this, it provides a decent abstraction
over the various virtualisation systems you may wish to use (virtualbox,
kvm, docker - we're only using virtualbox internally as it's the most
widely supported provider in Vagrant) - setting up the required networking,
mapping filesystems into VMs & managing /etc/hosts on guests & on the host
as required.

However, I'm not sure Vagrant provides a good solution for testing puppet
modules in isolation (I think it's great for the
system-config/project-config scenario, where you want to see how applying
the full set of required puppet modules on to an empty VM provides a
working system), it's harder to test standing up zuul without also setting
up a few other components, so puppet-zuul (for example) may not take
advantage of Vagrant directly, but may benefit from beaker[1] or
test-kitchen[2] work (I think that conversation has happened before but I
wasn't directly involved at the time)

[1] https://github.com/puppetlabs/beaker
[2] https://github.com/neillturner/kitchen-puppet & http://kitchen.ci/
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

[James E. Blair]

Joshua Hesketh  writes:

> There's probably further discussions here but I don't have enough knowledge
> in this area to comment further. The aim though should be to make it easy
> to bootstrap a server with the module you're developing on so you can
> easily verify and debug your changes.

Don't forget the documentation we already have on this subject:

http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Development environments for infra's puppet modules

[Joshua Hesketh]

Hey all,

We should look for a way to make developing, debugging and testing our
puppet modules locally easier and more consistent.
Short of bootstrapping an entire clone of openstack-infra, how do
developers currently set up an environment to investigate how a puppet
module behaves? This brings me to my first question:
1) Do we want to find/provide a way to set up a consistent development
environment?
Vagrant could be a useful way of providing a consistent development
environment for those working on infra's puppet modules. This comes up in
light of https://review.openstack.org/#/c/355273 which was split out from a
larger change due to debate over any vagrant precedent. This change was in
turn based upon this documented example of simulating OpenStack Infra
environments for testing:
https://krotscheck.net/2016/06/01/how-to-simulate-an-openstack-infra-slave.html
Currently the only module (that a quick grep found for me) providing a
vagrant file is puppet-storyboard.
2) Is Vagrant a good fit for this? Otherwise should we consider an
ansible-playbook to bootstrap an environment?
3) Where do we store and document such procedures (for example, in the
puppet repos themselves, as a guide somewhere, links to pastebin scripts
etc)
There's probably further discussions here but I don't have enough knowledge
in this area to comment further. The aim though should be to make it easy
to bootstrap a server with the module you're developing on so you can
easily verify and debug your changes.
Thoughts?
Cheers,Josh
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra