[Puppet Users] Location of config files in roles/profiles pattern

[J.T. Conklin]

Over the past year or so, we've completed a transition from locally
developed puppet modules containing node-specific configuration, to
using the roles/profiles pattern with parameterized modules with most
config values coming from hiera.

One exception to this are config files that are too specific, to
complicated, or otherwise not suitable for conversion to templates.  At
the moment, these are still in our puppet modules.  Not only does this
bind node-specific configuration in with otherwise independent modules,
now that all our other node-specific configuration is done with hiera,
the config is split across two places -- which makes it hard to under-
stand.

I spent some time look for articles and blog posts that cover this, but
all the examples I've seen show use cases where module configuration is
completely taken from hiera. How do others handle this?  Store files in
the profiles module itself?

Thanks in advance,

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87a856sbhk.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] PE: Updating custom SSL certs

[J.T. Conklin]

I'm running PE 2016.4.3 in my home lab. About 90 days ago, I installed
SSL certificates I generated via Let's Encrypt for my puppet server as
described @ https://docs.puppet.com/pe/2016.4/custom_console_cert.html.

With the cert expiration near, I re-generated my certs, updated the
public-console.cert.pem and public-console.private_key.pem files in
/opt/puppetlabs/server/data/console-services/certs, and kicked off
puppet -- and nothing happened.  Is something else needed to make the
updated certs take effect?

   --jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87bmqqj12g.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Can PE ENC variables be used in hiera hierarchy definition?

[J.T. Conklin]

I'm one of the admins of a site running PE 2016.4.2.  Over the last
year, we've migrated from our own puppet module organization to the
roles and profiles pattern.

Currently, we're setting the parameters for modules that are shared
by more than one role/profile on a per-node basis. This has a draw-
back that we have many node *.yaml files for roles with many hosts;
and we must remember to add addiitonal node *.yaml files whenever
new hosts are assigned to the role.

It would be convienent if we could have role-based *.yaml files. If
we create a 'role' variable in the PE ENC for each classification
group, can that be used in the hierarchy definition in hiera.yaml?
If so, what would happen for unclassified nodes where the new 'role'
variable is not defined?

Thanks,

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87d1ep7vam.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] r10k, git and .gitignore

[J.T. Conklin]

Robert  writes:
> Now there is just one question mark left*:
> - in some modules, under files, I store some bigger binaries (like
> apache-maven.tgz or a jdk-1.8.tar.gz) for which I'd like to use .
> gitignore files in order to leave these out of the repository, since
> they would consume storage space and are easy to download again so
> they don't need to be tracked. How could these files still be included
> in the files directory after the r10k run?

At my day job we had a few modules that did the same.  We addressed
this by moving the tarballs out of .../files, creating *.rpm and *.deb
packages with fpm (https://github.com/jordansissel/fpm), hosting those
packages in our local package manager (we use JFrog's Artifactory for
development, so it was convenient and available for IT use), and then
using puppet's package resource to manage package installation.

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87bmwequzr.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetfile git url representation

[J.T. Conklin]

Rob Nelson  writes:
> J.T., is it possible that the shell Git was using a key on disk that
> was removed recently?

That's my working hypothesis. Just haven't escaped from a constant
stream of other interrupts in order to verify.

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87a8f1gde4.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Puppetfile git url representation

[J.T. Conklin]

At work, I'm taking over primary administration of a puppet deployment
that had previously been maintained by a contractor. Shortly before he
left, it upgraded from PE 3.X to 2016.2.0, and moved from r10k to Code
Manager.

This has been running well, but "something happened" yesterday where
code deploy started failing with:

ERROR-> Unable to determine current branches for Git source 'puppet'
(/etc/puppetlabs/code-staging/environments)
Original exception:
Git remote "ssh://g...@stash.example.com/puppet/puppetcode.git" uses
the SSH protocol but no private key was given at

/opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-1/ssh---...@stash.example.com-ro-puppetcode.git

Going through the PE documentation, I discovered that a SSH private key
wasn't set up (suggesting that shell git had been being used instead of
rugged?), so I set puppet_enterprise::profile::master::r10_private_key
and forced a puppet run.

After that, deploys could fetch our control repository, but failed for
all our own modules in separate git repositories with a "Invalid
credential type"..

016-09-20 14:13:48,587 ERROR [deploy-pool-1] [p.c.core] Errors while
deploying environment 'dev' (exit code: 1):
INFO -> Deploying environment
/etc/puppetlabs/code-staging/environments/dev
INFO -> Environment dev is now at
f39d43d8e029c3208c98a76bf019387fa5cfa848
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/ntp
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/apt
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/concat
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/stdlib
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/sysctl
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/java
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/autofs
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/puppet_agent
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/transition
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/inifile
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/acpica_tools
ERROR-> Invalid credential type
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/apache
ERROR-> Invalid credential type
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/appfirst
ERROR-> Invalid credential type
INFO -> Deploying module
/etc/puppetlabs/code-staging/environments/dev/modules/aspell
ERROR-> Invalid credential type

One difference from the PE documentation (which I didn't think could
possibly be the problem, but fortunately tried anyway) was that our
Puppetfile git urls were of the form:

   ssh://g...@stash.example.com/puppet/puppet-foo

instead of

   g...@stash.example.com:puppet/puppet-foo

Once I changed Puppetfile to use the second form, I was again able to
use puppet-code to deploy changes. While I still need to find out the
"something" that happened to break things, I wanted to send a message to
the list while the details are still fresh in my mind in case this might
be useful to someone sometime down the road, especially as this behavior
(IMHO) violates the principle of least astonishment.

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87intpi3uh.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] mysql package name

[J.T. Conklin]

Suresh Rajagopal  writes:
> version is passed via hiera. params.pp has the logic to decide the
> package name based version. Is it a good practice to have this logic
> in params.pp ?

The philosophy I use in my own modules is that params.pp should provide
the best possible default based on the host OS version. But at the same
time it should be a parameter, so it can be overriden by hiera to handle
cases where that default is wrong.

It's very rare when I have to do that. And when I do, it's usually only
needed until params.pp can be updated to handle the case (e.g., when a
package name has changed in a new version of an OS, and the module has
not been updated to handle that new version).

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87k2ezi4vj.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Patterns for multi-arch libraries...

[J.T. Conklin]

Rob Nelson  writes:
> Silly question, but what package manager doesn't let you upgrade those
> two packages independently but also doesn't update the dependent
> packages at the same time?

We have this problem on CentOS machines using the yum provider. The logs
reported something to the effect of openssl.x86_64 couldn't be updated
to version N+1 as that conflicted with openssl.i686 version N.  I wish
I had saved the logs at the time so I could share the exact text with
you all.

For a while - when it seemed like there was a new OpenSSL vulnerabilty
every other day - we had the openssl module's "version" parameter set to
"latest" in our hiera config.  When a new openssl version was available,
puppet would attempt and fail to install it each run. We'd manually have
to install the new version - so much for saving time. I'm hoping to find
a better option before the next time we need to update.

   --jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/877fftd1lq.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Strategies for "boring" packages

[J.T. Conklin]

Luke Bigum  writes:
> In my mind the "purest" way would be to go individual modules for each
> package/service combination. If the only requirement is that you are
> handling the differences between Red Hat and Debian flavours, then a
> module per package/service. These modules would be wholly self
> contained and rely on some of the standard set of Facter facts. And
> then you could publish them :-) It would also avoid future duplicate
> resource declarations where someone's embedded "packageX" into one
> profile, and it clashes with "packageX" in another profile.
>
> I can see the argument for putting package installs and service starts
> into a profile but only if it's global for every operating system. So
> if there was profile::webserver that needed Package[openssl] and that
> was correct for all operating systems, then fine. However if you have
> to start doing conditional logic to find the right name of
> Package[openssl] for Red Hat and Debian, then profile::webserver is
> not the place. profile::webserver is a container of business logic
> that relates wholly and only to your business and your team. The exact
> implementation of Package[openssl] has nothing to do with
> profile::webserver, as long as openssl gets there somehow, that should
> be all you care about at the Profile level. Implementing
> Package[openssl] really depends on the operating system Facts alone,
> and this should be in it's own module... and... all of a sudden your
> profile::webserver is operating system agnostic, which is cool.

I agree 100%.

> Question - why is it taking your team getting annoyed at generating
> boilerplate code? Surely you have some sort of "puppet module create"
> wrapper script or you use https://github.com/voxpupuli/modulesync? If
> you've got so much overhead adding boiler plate code for your boring
> modules then I think you're tackling the wrong problem... If you can
> bring the boiler plate code problem down to 1-2 minutes, it's got to
> only take another 5-10 minutes tops to refactor one package{} and one
> service{} resource out of the profile and into it's own module, and
> then your team argument kind of goes away.

Lately we've been creating a new module every 3-4 weeks. So it's been
faster to copy an existing module, run a perl script that renames the
module, packages, and services, than it would be to write/adapt a script
to generate new modules from a template + parameters. It only takes me a
minute or two to create a new module. The counter-argument is that it
only takes a few seconds to add a "package { 'foo': }" to a profile, and
that a module per package/service leads to a unmanagable set of hundreds
of modules.

While I'm in the camp that separate modules for each package/service are
a good thing, I started this thread in order to double check my opinion.

> Question - why are you writing 120 modules yourself? Are there really
> no other implementations of these things on the Forge or GitHub?

In some cases we've found existing modules, we even use a few.  But in
the general case, we've found it useful to write our own modules so they
have the same "look and feel" i.e. use the same sets of parameters and
facts with the same semantics. Our basic package/service boilerplate is
based off the example42 modules (at least as they were ~2-3 years ago).

   --jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87bn55d2bj.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Patterns for multi-arch libraries...

[J.T. Conklin]

Before we started using puppet at work, we didn't have a systematic
process to install packages on our servers.  As a result, we had servers
in the same "role" that (unintentionally) had different sets of packages
installed.

A very common case was that the 64 bit version of a library was
installed but the corresponding 32 bit version was not.  This usually
wouldn't be detected until one of our users tried to run a 32 bit binary
that used that library on that particular system.

So we used puppet to bring our systems in line, selecting what packages
to install based on architecture.  For example, in our openssl module,
we changed:

  $package = 'openssl'

to:

  if $architecturee == 'x86_64' {
$package = [ 'openssl.x86_64', 'openssl.i686' ]
  } else {
$package = 'openssl'
  }

While this worked the first time to install the 32 bit openssl on all 64
bit machines, it fails when installing/updating to a new openssl version
(due to security issues, etc. as has been necessary recently) as the yum
provider attempts to install the 64 and 32 bit packages in the array
separately when they must be installed simultaneously.  I tried changing
the array to:

 $package = [ 'openssl', 'openssl.x86_64', 'openssl.i686' ]

thinking that having the un-suffixed version first would cause both to
be updated in the common case, and the suffixed versions to install the
specific 32 or 64 bit version if it's not already installed.
Unfortunately, this fails the same way.

Now that all our legacy systems have been "fixed", I suppose we could
drop the package array and go back to the original un-suffixed package
definition. But if there is a pattern that will work, I'd prefer using
it instead. Do any of you have a working idiom?

Thanks in advance,

--jtc



-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87oa96o0yu.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Strategies for "boring" packages

[J.T. Conklin]

At work, we've written about 120 modules in our puppet code repository.
About two dozen are "interesting", in that they have lots of parameters
and configuration that is specific to our environment.  The balance are
"boring", rather they are mostly boilerplate with minimal configuration.
For example, our modules abstract the differences in package and service
names between RedHat and Debian based systems.

However, there is some disagreement amongst our puppeteers about how to
handle these "boring" modules. One side objects to the amount of boiler-
plate and duplication, and would prefer that we simply define packages
in our role/profile modules. The other side claims that abstracting
package and service names is value enough to justify the overhead, and
that "boring" packages often become "interesting" over time as new
requirements for flexibility and customization develop over time. Each
group is firmly convinced that their opinion is the right one.

So I throw the question to the puppet community... What strategies do
you use for "boring" modules so you're not overwhelmed by hundreds of
small boilerplate modules?

Thanks for sharing,

--jtc

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/87shyio252.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.