Sounds like a great topic for the puppet operator session... one of the CERN people can explain how we're now doing it (since we're moving to a new structure after 12 months experience). Cells makes it even more interesting.
Tim > -----Original Message----- > From: Jonathan Proulx [mailto:[email protected]] > Sent: 03 October 2014 18:01 > To: Michael Dorman > Cc: [email protected] > Subject: Re: [Openstack-operators] Nodes and configurations management in > Puppet > > We (http://www.csail.mit.edu) have a very complex (chaotic) puppet world > outside of OpenStack as we have dozens of research groups, each with servers > and workstations that all need slightly different though mostly the same > things > along with "default" configs for these classes of systems and the various > infrastructure services that actually run it all. > > This gives us a deeper-than-I'd-like hiera hierarchy, though only the top > three > are really used for OpenStack specific stuff: > > :hierarchy: > - %{fqdn} > - %{role} > - %{cluster} > - %{group} > - %{lsbdistcodename} > - %{osfamily} > - common > > $cluster is where we define "production", "test" etc, > > $role specifices "compute-node" or "controller". > > $fqdn is rarely used, but we maintain a testing host-aggregate for last stage > tests in the production cloud occasionally teh set of fqdn.yaml files that > match > the hypervisors in that aggregate get symlinked to some hiera data before > actually putitng it in the $role.yaml that all the hypervisors get. I'm > currently > using this to verify my ceph integration will really & truly work for > ephemeral > storage... > > Though I'm also still using the deprecated > https://github.com/stackforge/puppet-openstack.git module last updated for > havana on my icehouse cloud, so take that as you will (the actual heavy > lifting > mods line 'nova' are newer) > > -Jon > > On Thu, Oct 2, 2014 at 11:50 PM, Michael Dorman <[email protected]> > wrote: > > We maintain a fairly flat hiera structure, which largely is due to our > > OS infrastructure still being pretty simple. > > > > Like Clayton & Matt, we use a ³world² attribute to indicate dev/test/prod. > > (Although in hindsight, I like the Œechelon¹ term a lot better. We > > did the same exercise of thinking of synonyms for Œenvironment.¹) So > > the structure looks like: > > > > - %{::world}/%{::clientcert} > > - %{::world} > > - global > > > > > > The global file is empty, and almost all of the config is stored in > > the world file. Over time, this has led to hiera sprawl so the world > > files have gotten quite messy. And there is a lot of items that > > aren¹t unique across worlds, so should really be in a global file. > > But, at the same time, this gives us a [mostly] single source of truth > > and avoids the ³grep -R² issue Joe described. > > > > ENC at this point is done by specifying a Œrole¹ parameter in the > > individual clientcert file for each node. This is a major downside, > > and doesn¹t scale, so we need to figure out something better. Maybe > > we can come up with a hostname scheme to encode the info there, like > > others have done. > > > > We run all masterless, for a variety of reasons (which limits ENC > > options, > > too.) Ansible is used to kick off runs across the environment. r10k > > deploys the Puppet environments (Œmaster¹ and Œprod¹ which correspond > > to git branches), heira data, and all the modules. Hiera data is in a > > separate (private) git repo, but there¹s only a master branch there. > > > > I¹ve been a big fan of the role/profile model, too, and it¹s worked > > well for us. One thing I¹ve thought about is specifying a list of > > profile classes for each node or node type in hiera, rather than > > maintaining a mostly static role module. Then we can just > > hiera_include(), which is the method we use in site.pp to include the > > role class now. I¹d be interested in others thoughts on this idea. I > > can¹t really think of a compelling reason to switch, other than it¹s kind of > clever. > > > > Mike > > > > > > > > > > On 9/26/14, 12:03 PM, "Mathieu Gagné" <[email protected]> wrote: > > > >>Hi Joe, > >> > >>Your experience and story about Puppet and OpenStack makes me feel > >>like you are a long lost co-worker. :) > >> > >>On 2014-09-25 10:30 PM, Joe Topjian wrote: > >>> > >>> Hiera takes the cake for my love/hate of Puppet. I try really hard > >>> to keep the number of hierarchies small and even then I find it > >>> awkward sometimes. I love the concept of Hiera, but I find it can be > >>> unintuitive. > >> > >>Same here. The aspect I hate about Hiera is that files become very big > >>and unorganized very fast due to the quantity of configs. So you try > >>to split them in multiples files instead and then you have the problem > >>you describe below... > >> > >> > >> > Similar to the other replies, I have a "common" hierarchy > >>> where 90% of the data is stored. The other hierarchies either > >>> override "common" or append to it. When I need to know where a > >>> parameter is ultimately configured, I find myself thinking "is that > >>> parameter common across everything or specific to a certain location > >>> or node, and if so, why did I make it specific?", then doing a "grep > >>> -R" to find where it's located, and finally thinking "oh right - that's > >>> why it's > there". > >> > >>Yep. That's the feeling I was referring to when I said "heart attack". > >> > >>And now, try to form a new co-worker and explain him how it's organized: > >>"Oh, I felt the file was too big so I split it in a hope to restore > >>sanity which it did with limited success." > >> > >>The other difficulty is the management of "common" configs like > >>keystone auth URL. Multiple services need this value, yet their might > >>be split in multiple files and the YAML anchor hack [1] I used so far > >>does not work across YAML files. Same for database configs which are > >>needed by the database server (to provision the user) and services > >>(for the database connection string). > >> > >> > >>> Another area of Puppet that I'm finding difficult to work with is > >>> configuring HA environments. There are two main pain points here and > >>> they're pretty applicable to using Puppet with OpenStack: > >> > > >>> The other HA pain point is creating many-to-one configurations [...] > >>> > >>> I think a cleaner way of doing this is to introduce service > >>> discovery into my environment, but I haven't had time to look into > >>> this in more detail. > >> > >>I wholly agree with you and that's a concept I'm interested to explore. > >>Come to think of it, it strangely looks like the "dependency inversion > >>principle" in software development. > >> > >>I however feel that an external ENC becomes inevitable to achieve this > >>ease of use. Unfortunately, each time I looked into it, I rapidly get > >>lost in my dream of a simple dashboard to manage everything. I feel I > >>rapidly come to the limits of what exported resources, Hiera and > >>puppetdb can do. > >> > >>One idea would be to export an haproxy::listen resource from one of > >>the controller (which now becomes a pet as you said) and realize it on > >>the HAProxy nodes with its associated haproxy::member resources. > >> > >> > >>> I should mention that some of these HA pains can be resolved by just > >>> moving all of the data to the HAProxy nodes themselves. So when I > >>> want to add a new service, such as RabbitMQ, to HAProxy, I add the > >>> RabbitMQ settings to the HAProxy role/profiles. But I want HAProxy to be > "dumb" > >>> about what it's hosting. I want to be able to use it in a Juju-like > >>> fashion where I can introduce any arbitrary service and HAProxy > >>> configures itself without prior knowledge of the new service. > >> > >>Yes! How do you guys think we can implement such discovery? > >> > >>With Nova cells, this problem became much more apparent due to > >>inter-relations between the API cell and compute cells. The API cell > >>has to know about the compute cells and vice versa. > >> > >> > >>> In general, though, I really enjoy working with Puppet. Our current > >>> Puppet configurations allow us to stand up test OpenStack > >>> environments with little manual input as well as upgrade to newer > >>> releases of OpenStack with very little effort. > >> > >>Yes, I really enjoy Puppet too. After all hardware/infrastructure > >>aspects are figured out, we are able to bootstrap a new OpenStack > >>region in less than an hour. > >> > >>To summarize my current pain points: > >>- Out of control Hiera configuration files > >>- Lack of service auto-discovery > >> > >>[1] https://dmsimard.com/2014/02/15/quick-hiera-tips/ > >> > >>-- > >>Mathieu > >> > >>_______________________________________________ > >>OpenStack-operators mailing list > >>[email protected] > >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator > >>s > > > > > > _______________________________________________ > > OpenStack-operators mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator > > s > > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
