Re: [Puppet Users] Re: Looking for solution on working configuration for new testing Puppet servers in existing environments
Hi Ken, Your site and your practices (and the problems that go with it) are similar to our own. I've answered by describing what we think the way forward for Puppet is for us, hopefully you find it helpful and get some ideas :-) I feel your pain when you say there are changes languishing in Dev for too long that get inadvertently pushed out to other environments. Once I accidentally rolled out a whole set of 'work in progress' changes that a colleague of mine was committing and pushing to one environment (we use Git) and I cut a new tag and pushed a whole heap of these unfinished changes to the next environment up the chain. After this I realised our current model really wasn't going to work in the long run: this happened with only two Admins working on Puppet in the last week when we employ almost a dozen who could potentially make the same mistakes. While we're still stuck in this model now, we think the way forward for us is to take a page out of our Development Team's book and set up a fully fledged CI environment for Puppet. It would be even more awesome if our Developer's CI was also our CI - but considering they've got half a million unit tests and we've got zero, we've got a long way to go. How this plays out practically is that all our admins commit into our Puppet repository and those changes start getting applied immediately to our Dev/CI servers and they start getting tested against immediately (hence the unit tests). Problems will always occur, but they will occur almost immediately and we'll have feedback to fix it. In response to your reply to Trevor: Can you explain how you see maintaining three slightly different environments with one Puppet Master and a firewall is more difficult than three individual Puppet Masters? You've already got all the differences written down in your existing manifests - different IP addresses for different services, different users/passwords, etc, maybe slightly different packages, or am I missing something? Without knowing more about your site (so hopefully this suggestion is relevant), here's what I would do: I would slowly start to encourage people to separate out your data from your code using a shared repository. Take a look at Hiera if you haven't already, I'd recommend using it but other data sources would suffice as well. So right now you've got your Dev data in your Dev modules, your Staging data in your Staging modules, etc. Take one of your modules (lets say DNS, it's easy), and put it's data (DNS server IP addresses) into a shared data store that each of your Dev, Staging and Prod Puppet Masters can access. Then change your Dev module to remove the data and pull it from the data store. Then do Staging, the same data store but different information, then finally Prod. If your three separate DNS modules don't look *exactly* the same now, you've done something wrong. Do that for every module and you've now got no reason not to run a single Puppet Master. The changes would take some time to do every module but if you encourage every admin to make a small step forward each change they make, eventually you'd get there. What kind of changes are you making 5-10 times a day? The same changes for the same people? We have a situation here where our Developers sometimes need to change the configuration of our application every hour. Right now we don't manage those systems with Puppet because making that many changes of the same thing over and over is a monumental waste of our time. I hack on Hiera a little bit to get some extra functionality from it, mainly the ability to tier my data sources. What I plan on doing is to have a low priority data store that's managed by Developers. Since all our modules will (eventually) look to the data stores to configure themselves, our Developers will be able to make their own system configuration changes. Our Admins control the modules (we write *how* it's done) and the higher priority data stores so we can override whatever the Developers enter if need be (we don't want a Developer changing the root password now). The Developers then trigger their own Puppet run and they've reconfigured the system using the same method (same Puppet modules) as Production so I don't have to worry about things being done differently in Dev. -Luke On 18/04/12 21:08, Ken Lareau wrote: Trevor, Thank you for the response; I believe you got the idea pretty well and while your suggestion makes sense, it is something we definitely can't follow through with right now; our configuration is massive and complex and having to maintain three different yet similar sets of configuration would be difficult and reduce our response time to necessary user changes (of which we get anywhere from 5-10 a day). It's just not feasible without a complete reworking of how we do things right now, and not at the top of our priority lists. I do appreciate the input, however. Thank
[Puppet Users] Re: Looking for solution on working configuration for new testing Puppet servers in existing environments
Hi Ken, thanks for posting. It seems like you have introduced some tension between the security requirements (clients which are in a particular environment must not be able to retrieve other environments) and the need to have widespread testing with good coverage. From what I understand you've managed this now by having different CA certificates for each environment, but -- as it sounds like you realise -- that is pretty problematic. I'd suggest you can end-run around a lot of this trouble by using an external node classifier to set and enforce client environment, so regardless of what the --environment string says from the client, each of the masters will consistently enforce policy. You don't have to go all-in to node class/parameter assignment in the ENC because static configs are merged with the output. A prerequisite is that you need to have some source of truth which the ENCs can consult to determine the disposition of each node that connects, but that source-of-truth could be an access-controlled database (maybe you already have one?) that is, in general, going to be a better place to put business logic (which nodes should be able to access each environment, and even perhaps some of the conditional logic in your manifests) than Puppet itself. This isn't without its own set of problems and might not be a panacea but I think would help a lot of your use case. Some relevant reading: http://docs.puppetlabs.com/guides/external_nodes.html https://projects.puppetlabs.com/issues/3910 https://projects.puppetlabs.com/issues/12869 Hope this helps - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook - On Tuesday, April 17, 2012 7:34:43 PM UTC-7, Ken Lareau wrote: Hello folks, After some conversation on #puppet on Freenode IRC, Eric Sorenson requested I repost the information and question here, so I am doing so and hopefully it will all make sense... We currently have a well-established and relatively complex Puppet setup in place at my company and I'm in the process of trying to streamline changes as well as implement better testing to ensure minimal disruption or issues when making those changes. Some information on the current situation: - There are currently three environments: development, staging, production. These are controlled via the '--environment' setting for puppet in each client. All clients only belong to one environment and do not move between them. - We have a single Puppet configuration to manage all environments. Various conditional statements based on environment, application type, hostname, etc. control what each client receives for its configuration. - There are separate servers for each environment for security reasons (primarily sensitive information that can only exist in the production environment). - The Puppet configuration maintained via a Git repo, currently on a single branch. - Each person on the admin team checks out own copy of the repo, make changes, commits the changes, then updates each environment on the Puppet servers for the changes to take effect. There are several issues with this process, unfortunately: - Every so often a configuration mistake will adversely affect an entire environment, and much of the time is only noticed _after_ the changes are pushed out. As a result, local changes tend to be made in the development environment for testing and sometimes aren't committed for a long time, leaving discrepancies between the environments which can lead to other subtle issues. - Less frequent but still occuring often enough, changes can still have subtle issues which cause things to work in one environment and break horribly in another; this is especially bad when the broken environment is the production one. - The configuration for a given type of client is complex enough that to change a client to a different application type (what we primarily key most of our configuration off of, followed by the environment) to test against a server would require rebuilding the client, which is a 25-45 minute process; too slow for simple changes and even too slow for all but the most complex changes, given how many changes we make in a single day. - We allow our users to create local VMs that the development Puppet server can key off their names to create a given configuration, but since the configuration for the various environments is shared in a Puppet configuration, potential for users point their puppet agents to the production environment is a concern (due to the sensitive information there). After discussion with a few coworkers, the following process was laid out to try to implement to resolve these issues: - Create separate branches for each of the environments and have only the matching branch checked out on the primary Puppet servers; changes will be merged into the various branches one at a time to prevent
[Puppet Users] Re: Looking for solution on working configuration for new testing Puppet servers in existing environments
I'll take a stab at some of this. Hopefully I'm correctly understanding your issue. Am I correct in the following? : You define 3 environments development, staging, and production. These environments are defined as such in Puppet but they are also separate environments within your network, for the sake of clarity I'll call them zones from here out? Each zone has a Puppet Master server. Each Puppet Master server has three environments defined development, staging, and production. Each environment has the full git repository with the applicable branch checked out. The clients in each zone connect to the Puppet Master in their zone and pull their configs from the corresponding environment. So a staging_zone_client connects to staging_zone_master and pulls from the staging_environment. If that's correct then: You already have three separate Puppet Masters so the environments are redundant. As configured staging_zone_client can pull from production_environment using --environment. One fix could be to define only the production environment on each zone's Puppet Master and check out the applicable branch in only the production environment. As long as you never check out the production branch in development or staging then clients from those zones couldn't pull the settings for production zone as it's just not available. As long as they cannot connect to the other zone's Puppet Master, preventable by network segmentation, certs etc... Within each zone you could then define environments such as development and testing for conducting those activities within each zone. So you'd have staging/dev and staging/test branches checked out in those environments. I guess you could extend that and create environments for each admin within each zone that would allow the admin to use the --environment option for clients to test their work within a zone. This would result in a lot of environments, and probably a lot of branches, but you wouldn't need a test Puppet Master for each admin. I'd think this would introduce the problem of making it difficult to reuse modules between zones as I'd think you'd end up basically managing three completely different branches. Unless the sensitive data you're worried about is not being stored in your puppet repo and you have no issues merging changes made to the production branch into the development and testing branches, plus your admins will have a lot of different topic branches to deal with. Long run you'd probably want to move zone specific settings out of your modules and use something like hiera so you can standardize your modules across zones and just pull in the location settings using hiera. Hope I understood your problems correctly and this is helpful.. On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote: Hello folks, After some conversation on #puppet on Freenode IRC, Eric Sorenson requested I repost the information and question here, so I am doing so and hopefully it will all make sense... We currently have a well-established and relatively complex Puppet setup in place at my company and I'm in the process of trying to streamline changes as well as implement better testing to ensure minimal disruption or issues when making those changes. Some information on the current situation: - There are currently three environments: development, staging, production. These are controlled via the '--environment' setting for puppet in each client. All clients only belong to one environment and do not move between them. - We have a single Puppet configuration to manage all environments. Various conditional statements based on environment, application type, hostname, etc. control what each client receives for its configuration. - There are separate servers for each environment for security reasons (primarily sensitive information that can only exist in the production environment). - The Puppet configuration maintained via a Git repo, currently on a single branch. - Each person on the admin team checks out own copy of the repo, make changes, commits the changes, then updates each environment on the Puppet servers for the changes to take effect. There are several issues with this process, unfortunately: - Every so often a configuration mistake will adversely affect an entire environment, and much of the time is only noticed _after_ the changes are pushed out. As a result, local changes tend to be made in the development environment for testing and sometimes aren't committed for a long time, leaving discrepancies between the environments which can lead to other subtle issues. - Less frequent but still occuring often enough, changes can still have subtle issues which cause things to work in one environment and break horribly in another; this is especially bad when the broken environment is the production one. - The configuration for a given
Re: [Puppet Users] Re: Looking for solution on working configuration for new testing Puppet servers in existing environments
Eric, Thank you for the response, and yes, our current configuration and security requirements have made things a bit difficult at the moment. Fortunately we do already have an ENC which does access an access-controlled database and does have the environment information in it, though we still do pass '--environment' due to this not working long ago... and looking at the first issue ticket you mention, this could be a problem as our developers are allowed to connect their own VMs to the development Puppet server but can easily choose to point '--environment' to whatever they please. In actuality, they can do that now (and it would probably work), so the problem is already there, though I would definitely not want to make it worse. :) Right now our ENC only has a minimal amount of information in it, but the plan is to eventually populate it with more and reduce the amount of work the Puppet configuration has to do itself, as you suggested below. From the brief exchange on IRC this morning, the indication seems to be moving to a single CA should be the follow-up to this, and I think this is doable, though I'm still uncertain what the best path is to handle this. Once that's done, I can then look into improving the security of our systems (as in actually making it secure rather than what it really is right now). Thank you for your input. - Ken Lareau On Wed, Apr 18, 2012 at 7:56 AM, Eric Sorenson eric.soren...@me.com wrote: Hi Ken, thanks for posting. It seems like you have introduced some tension between the security requirements (clients which are in a particular environment must not be able to retrieve other environments) and the need to have widespread testing with good coverage. From what I understand you've managed this now by having different CA certificates for each environment, but -- as it sounds like you realise -- that is pretty problematic. I'd suggest you can end-run around a lot of this trouble by using an external node classifier to set and enforce client environment, so regardless of what the --environment string says from the client, each of the masters will consistently enforce policy. You don't have to go all-in to node class/parameter assignment in the ENC because static configs are merged with the output. A prerequisite is that you need to have some source of truth which the ENCs can consult to determine the disposition of each node that connects, but that source-of-truth could be an access-controlled database (maybe you already have one?) that is, in general, going to be a better place to put business logic (which nodes should be able to access each environment, and even perhaps some of the conditional logic in your manifests) than Puppet itself. This isn't without its own set of problems and might not be a panacea but I think would help a lot of your use case. Some relevant reading: http://docs.puppetlabs.com/guides/external_nodes.html https://projects.puppetlabs.com/issues/3910 https://projects.puppetlabs.com/issues/12869 Hope this helps - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook - On Tuesday, April 17, 2012 7:34:43 PM UTC-7, Ken Lareau wrote: Hello folks, After some conversation on #puppet on Freenode IRC, Eric Sorenson requested I repost the information and question here, so I am doing so and hopefully it will all make sense... We currently have a well-established and relatively complex Puppet setup in place at my company and I'm in the process of trying to streamline changes as well as implement better testing to ensure minimal disruption or issues when making those changes. Some information on the current situation: - There are currently three environments: development, staging, production. These are controlled via the '--environment' setting for puppet in each client. All clients only belong to one environment and do not move between them. - We have a single Puppet configuration to manage all environments. Various conditional statements based on environment, application type, hostname, etc. control what each client receives for its configuration. - There are separate servers for each environment for security reasons (primarily sensitive information that can only exist in the production environment). - The Puppet configuration maintained via a Git repo, currently on a single branch. - Each person on the admin team checks out own copy of the repo, make changes, commits the changes, then updates each environment on the Puppet servers for the changes to take effect. There are several issues with this process, unfortunately: - Every so often a configuration mistake will adversely affect an entire environment, and much of the time is only noticed _after_ the changes are pushed out. As a result, local changes tend to be made in the development environment for testing and sometimes aren't committed for a long time, leaving discrepancies between the environments which can lead
Re: [Puppet Users] Re: Looking for solution on working configuration for new testing Puppet servers in existing environments
Trevor, Thank you for the response; I believe you got the idea pretty well and while your suggestion makes sense, it is something we definitely can't follow through with right now; our configuration is massive and complex and having to maintain three different yet similar sets of configuration would be difficult and reduce our response time to necessary user changes (of which we get anywhere from 5-10 a day). It's just not feasible without a complete reworking of how we do things right now, and not at the top of our priority lists. I do appreciate the input, however. Thank you. - Ken Lareau On Wed, Apr 18, 2012 at 12:28 PM, Trevor Smith trevor.c.sm...@gmail.comwrote: I'll take a stab at some of this. Hopefully I'm correctly understanding your issue. Am I correct in the following? : You define 3 environments development, staging, and production. These environments are defined as such in Puppet but they are also separate environments within your network, for the sake of clarity I'll call them zones from here out? Each zone has a Puppet Master server. Each Puppet Master server has three environments defined development, staging, and production. Each environment has the full git repository with the applicable branch checked out. The clients in each zone connect to the Puppet Master in their zone and pull their configs from the corresponding environment. So a staging_zone_client connects to staging_zone_master and pulls from the staging_environment. If that's correct then: You already have three separate Puppet Masters so the environments are redundant. As configured staging_zone_client can pull from production_environment using --environment. One fix could be to define only the production environment on each zone's Puppet Master and check out the applicable branch in only the production environment. As long as you never check out the production branch in development or staging then clients from those zones couldn't pull the settings for production zone as it's just not available. As long as they cannot connect to the other zone's Puppet Master, preventable by network segmentation, certs etc... Within each zone you could then define environments such as development and testing for conducting those activities within each zone. So you'd have staging/dev and staging/test branches checked out in those environments. I guess you could extend that and create environments for each admin within each zone that would allow the admin to use the --environment option for clients to test their work within a zone. This would result in a lot of environments, and probably a lot of branches, but you wouldn't need a test Puppet Master for each admin. I'd think this would introduce the problem of making it difficult to reuse modules between zones as I'd think you'd end up basically managing three completely different branches. Unless the sensitive data you're worried about is not being stored in your puppet repo and you have no issues merging changes made to the production branch into the development and testing branches, plus your admins will have a lot of different topic branches to deal with. Long run you'd probably want to move zone specific settings out of your modules and use something like hiera so you can standardize your modules across zones and just pull in the location settings using hiera. Hope I understood your problems correctly and this is helpful.. On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote: Hello folks, After some conversation on #puppet on Freenode IRC, Eric Sorenson requested I repost the information and question here, so I am doing so and hopefully it will all make sense... We currently have a well-established and relatively complex Puppet setup in place at my company and I'm in the process of trying to streamline changes as well as implement better testing to ensure minimal disruption or issues when making those changes. Some information on the current situation: - There are currently three environments: development, staging, production. These are controlled via the '--environment' setting for puppet in each client. All clients only belong to one environment and do not move between them. - We have a single Puppet configuration to manage all environments. Various conditional statements based on environment, application type, hostname, etc. control what each client receives for its configuration. - There are separate servers for each environment for security reasons (primarily sensitive information that can only exist in the production environment). - The Puppet configuration maintained via a Git repo, currently on a single branch. - Each person on the admin team checks out own copy of the repo, make changes, commits the changes, then updates each environment on the Puppet servers for the changes to take effect. There are several issues with this process, unfortunately: - Every
