[Puppet Users] Re: Banned from RHN due to exessive connections; would like opions on my solution.
On Wednesday, July 23, 2014 6:26:00 PM UTC-5, Stack Kororā wrote:
Greetings!
Thank you so much John. I just learned something new about Puppet. Utilizing
inline_template is a heck of a lot easier then how I first attempted that
variable substitution. I might have to go back and fix some of my older code
later
Here are a few other notes in response to your email:
Have you considered setting up a caching proxy between you and them?
We have discussed doing a caching proxy, but haven't ever had the
time/inclination to implement one yet.
What per-package request(s) is yum actually making?
I explored the yum thing a bit more. Running puppet-3.6.2-1.el6.noarch on
both server and client using CentOS6 as my test systems. I started a puppet
run in one terminal and ran this code in a second:
$ while [ $(pgrep puppet) != ]; do pgrep yum; done | uniq
If I just do this:
package { 'telnet' : ensure=absent,}
Nothing triggers.
If I do it this way:
$removethesepackages = [
'telnet-server',
'telnet',
]
package {$removethesepackages : ensure=absent,}
Then I get a yum PID per package. For every PID I get a line in the puppet
log like this:
Notice: /Stage[main]/audit::Software_disabled/Package[telnet]/ensure: created
(there is that weird error message again where an absent is created).
I don't know why. Both work as expected, but the second triggers a yum call
the first doesn't.
I find it surprising -- and in fact unlikely -- that using an array
resource title produces different behavior than does a sequence of separate
resource declaration for each of the packages. If that is in fact the case
then I encourage you to file a bug report, but I'm pretty sure you're mixed
up. In particular, Puppet will *never* invoke 'yum' to ensure a package
'absent' -- the 'yum' provider intentionally uses 'rpm -e' for that.
Puppet uses 'yum remove' only to ensure a package 'purged'.
In any case, the issue is not so much whether Puppet spawns yum processes;
rather it is what yum *does* once spawned. Yum should not be hitting RHN
to service a request to remove a package, whether that package is in fact
installed or not. Furthermore, the number of RHN requests needed to serve
all other package management needs should be minimized by yum's metadata
caching. Even if you instructed Puppet to clear yum metadata on each run,
you should hit RHN only once per Puppet run, except when you install or
update a package. If you see different behavior then you should
investigate yum's own configuration.
So I thought, 'Maybe it is hitting local cache and not actually going out to
the repo'. I dug around in the logs on our local repo and found this:
[IP REMOVED] - - [23/Jul/2014:14:07:58 -0500] GET
/puppetlabs/6/products/x86_64/repodata/repomd.xml HTTP/1.1 200 2529 -
urlgrabber/3.9.1 yum/3.2.29
It isn't one per package, but it is one per puppet run. Something about that
method calls yum differently I guess. Not sure why.
Puppet executes a yum list --all once per run so it can tell what to do
with packages ensured anything other than 'absent' or 'purged'. This is
part of a strategy to avoid needlessly running yum to install packages that
aren't available or to update a packages that are already at the latest
available version. But it should be served from yum's cache most of the
time. If you see these frequently going out to RHN, then again, something
is weird about your yum configuration.
---
The double notice I was referring to is this:
Notice: Package telnet is not installed
Notice:
/Stage[main]/audit::Software_disabled/audit::Forbidden_package[telnet]/Notify[Package
telnet is not installed]/message: defined 'message' as 'Package telnet is
not installed'
I am told three times in two lines (more with wrap around on a console) that
telnet isn't installed. I find it annoying and haven't found a solution to
removing it yet and leaving just the first Notice. If you know of one I would
be /very/ grateful.
So that's just what I supposed you might be talking about. Did you try my
suggestion to set loglevel = 'debug' on the Notice resources?
I implemented your code and it is working brilliantly. I made two changes.
1) I placed the define in init.pp so I can reference it anywhere in the audit
class easily.
Sorry, I didn't intend to imply a specific file to put the define in. The
standard place for a define named audit::forbidden_package would be module
path/audit/manifests/forbidden_package.pp (same rule as for classes).
Putting it there is even more likely to ensure that Puppet can always find
it. If you follow that pattern systematically then it also helps *you*
find it.
2) I changed:
'%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %')
to:
%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %)
Using the single quotes gave me the
[Puppet Users] Re: Banned from RHN due to exessive connections; would like opions on my solution.
On Tuesday, July 22, 2014 7:37:30 PM UTC-5, Stack Kororā wrote:
Greetings,
In my multiple hundred servers, I have 10 that are Red Hat based. We
recently brought them under the same management as the rest of the servers
utilizing Puppet. Then we ran into issues because we were hitting RHN too
frequently and we got our servers banned. :-(
I went digging for the culprit and found it in a section we wrote because
audit is insistent that some packages should never be installed and they
want regular checks that the packages are not installed. I rather liked my
original solution (below) as I have dozens of packages that shouldn't be
installed and occasionally I get another one to add to the list. This code
made it really simple to add a new package.
class audit::software_remove (
) {
$removethesepackages = [
'telnet-server',
'telnet',
# Dozens removed for sanity :-)
]
case $operatingsystem {
'SLES' : { package {$removethesepackages : ensure=absent,} }
'Scientific', 'CentOS', 'RedHat' : { package {$removethesepackages
: ensure=purged,} }
default : {}
}
}
Now SLES runs this code amazingly well because zypper just does a check
against the local install before trying to remove a package and if it isn't
installed it doesn't do anything at all. One of the many shortcomings of
yum is that it always hits the repos.
But that's what's curious about your issue: it's not normal for yum to hit
the repos to service every request. Yum (by default) maintains local
caches of repo metadata to avoid that. At least on CentOS. I am not
positive that *bona fide* RHEL has the same default, but it certainly has
the same capability.
Since we have a local repo set up for SLES, Scientific, and CentOS we
don't care if we beat up on them. However, we discussed the local repo
caching with Red Hat and it didn't work out (too much complexity for too
few servers; won't go into details).
Have you considered setting up a caching proxy between you and them?
Not only that, but because every package is a separate transaction, we hit
the repo /multiple/ times every puppet run. Thus, our servers hit Red Hat
repos frequently and we get banned. :-(
Again, this doesn't make sense. What per-package request(s) is yum
actually making? It does not even request repo metadata for me when I ask
it to remove a package that is not installed (much less any
package-specific data).
So I went thinking about how to solve both the problem of slamming the
repo and checking for the package before trying to do a removal.
We have and utilize the pkginventory module[1] (which we have applied the
waiting patches plus done some of our own since that module hasn't been
updated in a long while). This gives me a fact of pkg_telnet or
pkg_telnet_server if those packages are installed. So I decided to utilize
that and I came up with the code below.
[1] https://forge.puppetlabs.com/ody/pkginventory
class audit::software_remove (
) {
if $pkg_telnet_server {
case $operatingsystem {
'SLES' : { package { 'telnet-server': ensure=absent,} }
'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server'
: ensure=purged,} }
default : {}
}
}
else { notify {Package telnet-server is not installed:}}
#
if $::pkg_telnet {
case $operatingsystem {
'SLES' : { package { 'telnet': ensure=absent,} }
'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
ensure=purged,} }
default : {}
}
}
else { notify {Package telnet is not installed:}}
}
Hrm. It works, but it isn't a good solution in my opinion. Not at all. Too
much code is being duplicated here. Well, there is some clean up I can do
though now. Previously I used the case statement because audit wanted us to
do a yum purge if any of the packages are found installed but SLES
doesn't support purged and I don't really see a difference. That is a
pointless case statement in my opinion with far too much code duplication.
class audit::software_remove (
) {
if $pkg_telnet_server { package { 'telnet-server': ensure=absent,} }
else { notify {Package telnet-server is not installed:}}
#
if $pkg_telnet { package { 'telnet': ensure=absent,} }
else { notify {Package telnet is not installed:}}
}
How about this:
define audit::forbidden_package () {
$is_present = inline_template(
'%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %')
if $is_present {
package { $title: ensure = absent }
} else {
notify { Package $title is not installed: }
}
}
class audit::software_remove (
) {
$forbidden_packages = [
'telnet-server',
'telnet',
# Dozens removed for sanity :-)
[Puppet Users] Re: Banned from RHN due to exessive connections; would like opions on my solution.
Greetings!
Thank you so much John. I just learned something new about Puppet. Utilizing
inline_template is a heck of a lot easier then how I first attempted that
variable substitution. I might have to go back and fix some of my older code
later
Here are a few other notes in response to your email:
Have you considered setting up a caching proxy between you and them?
We have discussed doing a caching proxy, but haven't ever had the
time/inclination to implement one yet.
What per-package request(s) is yum actually making?
I explored the yum thing a bit more. Running puppet-3.6.2-1.el6.noarch on both
server and client using CentOS6 as my test systems. I started a puppet run in
one terminal and ran this code in a second:
$ while [ $(pgrep puppet) != ]; do pgrep yum; done | uniq
If I just do this:
package { 'telnet' : ensure=absent,}
Nothing triggers.
If I do it this way:
$removethesepackages = [
'telnet-server',
'telnet',
]
package {$removethesepackages : ensure=absent,}
Then I get a yum PID per package. For every PID I get a line in the puppet log
like this:
Notice: /Stage[main]/audit::Software_disabled/Package[telnet]/ensure: created
(there is that weird error message again where an absent is created).
I don't know why. Both work as expected, but the second triggers a yum call the
first doesn't.
So I thought, 'Maybe it is hitting local cache and not actually going out to
the repo'. I dug around in the logs on our local repo and found this:
[IP REMOVED] - - [23/Jul/2014:14:07:58 -0500] GET
/puppetlabs/6/products/x86_64/repodata/repomd.xml HTTP/1.1 200 2529 -
urlgrabber/3.9.1 yum/3.2.29
It isn't one per package, but it is one per puppet run. Something about that
method calls yum differently I guess. Not sure why.
---
The double notice I was referring to is this:
Notice: Package telnet is not installed
Notice:
/Stage[main]/audit::Software_disabled/audit::Forbidden_package[telnet]/Notify[Package
telnet is not installed]/message: defined 'message' as 'Package telnet is not
installed'
I am told three times in two lines (more with wrap around on a console) that
telnet isn't installed. I find it annoying and haven't found a solution to
removing it yet and leaving just the first Notice. If you know of one I would
be /very/ grateful.
I implemented your code and it is working brilliantly. I made two changes.
1) I placed the define in init.pp so I can reference it anywhere in the audit
class easily.
2) I changed:
'%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %')
to:
%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %)
Using the single quotes gave me the error:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Syntax error at '::pkg_'; expected ')' at
/etc/puppet/modules/audit/manifests/software_disabled.pp:8 on node
centos6.testing.puppet
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
But now it is working really well in my dev environment! I push to production
tomorrow...We will see how pleased I am with my code changes at the end of the
day after this fix + 4 other minor changes roll out. :-D
Thank you to everyone who has chimed in. These responses are exactly what I was
looking for. I have learned more about puppet and have a few new tricks to use.
I really do appreciate it.
Thanks!
~Stack~
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/1d09b25c-55ab-486b-a6c4-e31803b9b813%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
