Re: [Puppet Users] Nodes getting catalog with incorrect resource.
Hi John Thanks for the piece of information! I'm pretty sure (at least on the environment where the problem is happening) that everything is inside classes. I'll try and look through our custom modules to see if there are any resources outside of a class block. On Wednesday, July 6, 2016 at 1:22:55 AM UTC+12, jcbollinger wrote: > > > > On Monday, July 4, 2016 at 4:20:12 PM UTC-5, Gino Lisignoli wrote: >> >> NRPE Module: https://github.com/example42/puppet-nrpe >> >> If you have a look at the params.pp file, lines 67-74 you will see a >> block of code that sets the value of a variable depending on the operating >> system version. In my environment my operating system is ubuntu, and I >> would expect the value of the $package variable to end up as: >> nagios-nrpe-server. But what's happening is the $package variable ends up >> as 'nrpe' which doesn't exist. >> >> The NRPE Module doesn't do anything with exported resources >> >> Anyway, running that code locally works fine. From a puppet master I get >> the problem described. Which is very strange. I also tried disabling our >> puppetdb server but still had the same problem. >> >> So yesterday I managed to resolve the problem, the resolution is to >> disable puppet environment cache. >> >> I know the situation to make the problem happen (turn environment caching >> on), but I don't know why it is happening. When the environment cache is >> turned on I can clear the the cache using the puppet admin api, and then >> for one puppet run it will work fine, but (I'm assuming) after it is cached >> it is getting confused with other nodes generating/using the same cache but >> with different facts. >> >> > > This problem has the hallmarks of a top-scope resource declaration -- i.e. > one outside any class, node block, or defined type. These are allowed, and > they are sometimes reasonable in a site manifest, but they should never > appear in manifests that are not guaranteed always to be evaluated, such as > those in modules. The problem is that top-scope declarations anywhere in > the manifest set apply in principle to all nodes, so if they appear in > manifests that Puppet evaluates only conditionally, such as via the module > autoloader, then the catalog builder gets an inconsistent view of the > global declarations. > > The usual symptom of inappropriate top-level declaration is that resources > seem to leak from some catalogs to others. It is also possible for > resource parameters to leak under certain circumstances. The fact that > clearing the cache resolves the problem temporarily is a strong indication > that your problem lies in this area. > > If you do have a problem with top-level declarations, then it is probably > in one of your own modules, not in the Example42 module. Probably it's in > a module that uses the example42 module, but it could be one that uses it > indirectly. > > > John > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/a4f2d89c-a65a-46e7-b70d-e6a7f8065ea6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Nodes getting catalog with incorrect resource.
On Monday, July 4, 2016 at 4:20:12 PM UTC-5, Gino Lisignoli wrote: > > NRPE Module: https://github.com/example42/puppet-nrpe > > If you have a look at the params.pp file, lines 67-74 you will see a block > of code that sets the value of a variable depending on the operating system > version. In my environment my operating system is ubuntu, and I would > expect the value of the $package variable to end up as: nagios-nrpe-server. > But what's happening is the $package variable ends up as 'nrpe' which > doesn't exist. > > The NRPE Module doesn't do anything with exported resources > > Anyway, running that code locally works fine. From a puppet master I get > the problem described. Which is very strange. I also tried disabling our > puppetdb server but still had the same problem. > > So yesterday I managed to resolve the problem, the resolution is to > disable puppet environment cache. > > I know the situation to make the problem happen (turn environment caching > on), but I don't know why it is happening. When the environment cache is > turned on I can clear the the cache using the puppet admin api, and then > for one puppet run it will work fine, but (I'm assuming) after it is cached > it is getting confused with other nodes generating/using the same cache but > with different facts. > > This problem has the hallmarks of a top-scope resource declaration -- i.e. one outside any class, node block, or defined type. These are allowed, and they are sometimes reasonable in a site manifest, but they should never appear in manifests that are not guaranteed always to be evaluated, such as those in modules. The problem is that top-scope declarations anywhere in the manifest set apply in principle to all nodes, so if they appear in manifests that Puppet evaluates only conditionally, such as via the module autoloader, then the catalog builder gets an inconsistent view of the global declarations. The usual symptom of inappropriate top-level declaration is that resources seem to leak from some catalogs to others. It is also possible for resource parameters to leak under certain circumstances. The fact that clearing the cache resolves the problem temporarily is a strong indication that your problem lies in this area. If you do have a problem with top-level declarations, then it is probably in one of your own modules, not in the Example42 module. Probably it's in a module that uses the example42 module, but it could be one that uses it indirectly. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4fca4db5-e217-4f3e-950e-fa17a4de3f34%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Nodes getting catalog with incorrect resource.
NRPE Module: https://github.com/example42/puppet-nrpe If you have a look at the params.pp file, lines 67-74 you will see a block of code that sets the value of a variable depending on the operating system version. In my environment my operating system is ubuntu, and I would expect the value of the $package variable to end up as: nagios-nrpe-server. But what's happening is the $package variable ends up as 'nrpe' which doesn't exist. The NRPE Module doesn't do anything with exported resources Anyway, running that code locally works fine. From a puppet master I get the problem described. Which is very strange. I also tried disabling our puppetdb server but still had the same problem. So yesterday I managed to resolve the problem, the resolution is to disable puppet environment cache. I know the situation to make the problem happen (turn environment caching on), but I don't know why it is happening. When the environment cache is turned on I can clear the the cache using the puppet admin api, and then for one puppet run it will work fine, but (I'm assuming) after it is cached it is getting confused with other nodes generating/using the same cache but with different facts. On Tuesday, July 5, 2016 at 12:26:19 AM UTC+12, Luke Bigum wrote: > > Can you explain the symptoms of your problem a bit more, and link to the > NRPE module you are using if it's open source? > > You've described everything you've checked, but I still don't know exactly > what's wrong / failing and what you are expecting to see :-) > > Does the NRPE module collect exported resources? Could you be collecting > Redhat resources onto your ubuntu machine? > > There may be some level of catalog caching happening here, but to get just > the operating system Facts confused and nothing else seems unlikely. You > could try turn unlplug PuppetDB from your Master temporarily (storeconfigs > = false?) and see what catalog gets compiled then. The Master should also > have the latest Facts for every node on disk in YAML format here: > > ls -ld /var/lib/puppet/yaml/facts/$(hostname).yaml > > Assuming that's still the right path in Puppet 4. > > -- > Luke Bigum > Senior Systems Engineer > > Information Systems > > - Original Message - > From: "Gino Lisignoli" <glisi...@gmail.com > > To: "puppet-users" <puppet...@googlegroups.com > > Sent: Monday, 4 July, 2016 01:09:28 > Subject: [Puppet Users] Nodes getting catalog with incorrect resource. > > I'm having the strangest puppet problem I have ever seen: > > I'm trying to get a nrpe module working on some nodes (Ubuntu 12.04 and > 14.04), but both modules I have tried, seem to think the operating system > is Redhat. > > To explain this a little bit better: > > - Facter on all these nodes reports that the osfamily is Debian, and the > operatingsystem is Ubuntu, which is correct. > - When I check the reported facts for the nodes on the puppetserver they > are Debian and Ubuntu > - When I look at the catalog that gets compiled I can see that all my > other > resources seem to be correct, we have some osfamily/system code that > checks > the os and sets up apt/yum repositories. So I know that's working correct. > - When I r10k my entire environment to the nodes and run it locally with > 'puppet apply' there are no problems at all. > - Sometimes when I clear the environment cache, the problem will go away > for the first puppet run, but all future puppet runs have the same > problem. > - I've checked hiera, the only paramater for nrpe is to specify our ntp > server > > I'm running puppet 4.5.1, puppetsever 2.4.0 and puppetdb 4.1.0, we use > Foreman to classify our nodes with no smart paramaters etc. The only > classification it does is assign a role to a node. > > My only ideas are: > > - There's some some sort of catalog/resource cache that is getting > generated with the wrong nrpe resource. AFAIK this doesn't exist? As only > whole catalogs are cached on the puppetserver > - Somehow puppetdb might be invloved? I know puppetdb has > resources/catalogs on it. Does puppetserver use puppetdb as a cache for > resources/catalogs when generating a catalog for nodes? If so is there a > way to debug why it would be generating a catalog with an incorrect > resource for nodes? > > Anyone have any ideas how to debug this? > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users...@googlegroups.com . > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/889f44ad-1e
Re: [Puppet Users] Nodes getting catalog with incorrect resource.
Can you explain the symptoms of your problem a bit more, and link to the NRPE module you are using if it's open source? You've described everything you've checked, but I still don't know exactly what's wrong / failing and what you are expecting to see :-) Does the NRPE module collect exported resources? Could you be collecting Redhat resources onto your ubuntu machine? There may be some level of catalog caching happening here, but to get just the operating system Facts confused and nothing else seems unlikely. You could try turn unlplug PuppetDB from your Master temporarily (storeconfigs = false?) and see what catalog gets compiled then. The Master should also have the latest Facts for every node on disk in YAML format here: ls -ld /var/lib/puppet/yaml/facts/$(hostname).yaml Assuming that's still the right path in Puppet 4. -- Luke Bigum Senior Systems Engineer Information Systems - Original Message - From: "Gino Lisignoli" <glisign...@gmail.com> To: "puppet-users" <puppet-users@googlegroups.com> Sent: Monday, 4 July, 2016 01:09:28 Subject: [Puppet Users] Nodes getting catalog with incorrect resource. I'm having the strangest puppet problem I have ever seen: I'm trying to get a nrpe module working on some nodes (Ubuntu 12.04 and 14.04), but both modules I have tried, seem to think the operating system is Redhat. To explain this a little bit better: - Facter on all these nodes reports that the osfamily is Debian, and the operatingsystem is Ubuntu, which is correct. - When I check the reported facts for the nodes on the puppetserver they are Debian and Ubuntu - When I look at the catalog that gets compiled I can see that all my other resources seem to be correct, we have some osfamily/system code that checks the os and sets up apt/yum repositories. So I know that's working correct. - When I r10k my entire environment to the nodes and run it locally with 'puppet apply' there are no problems at all. - Sometimes when I clear the environment cache, the problem will go away for the first puppet run, but all future puppet runs have the same problem. - I've checked hiera, the only paramater for nrpe is to specify our ntp server I'm running puppet 4.5.1, puppetsever 2.4.0 and puppetdb 4.1.0, we use Foreman to classify our nodes with no smart paramaters etc. The only classification it does is assign a role to a node. My only ideas are: - There's some some sort of catalog/resource cache that is getting generated with the wrong nrpe resource. AFAIK this doesn't exist? As only whole catalogs are cached on the puppetserver - Somehow puppetdb might be invloved? I know puppetdb has resources/catalogs on it. Does puppetserver use puppetdb as a cache for resources/catalogs when generating a catalog for nodes? If so is there a way to debug why it would be generating a catalog with an incorrect resource for nodes? Anyone have any ideas how to debug this? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/889f44ad-1ece-4dff-a769-b91c916d4bee%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. --- LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN http://www.LMAX.com/ Recognised by the most prestigious business and technology awards 2016 Best Trading & Execution, HFM US Technology Awards 2016, 2015, 2014, 2013 Best FX Trading Venue - ECN/MTF, WSL Institutional Trading Awards 2015 Winner, Deloitte UK Technology Fast 50 2015, 2014, 2013, One of the UK's fastest growing technology firms, The Sunday Times Tech Track 100 2015 Winner, Deloitte EMEA Technology Fast 500 2015, 2014, 2013 Best Margin Sector Platform, Profit & Loss Readers' Choice Awards --- FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved. This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message. LMAX Exchange is the trading name of LMAX Limited. LMAX Limited opera
[Puppet Users] Nodes getting catalog with incorrect resource.
I'm having the strangest puppet problem I have ever seen: I'm trying to get a nrpe module working on some nodes (Ubuntu 12.04 and 14.04), but both modules I have tried, seem to think the operating system is Redhat. To explain this a little bit better: - Facter on all these nodes reports that the osfamily is Debian, and the operatingsystem is Ubuntu, which is correct. - When I check the reported facts for the nodes on the puppetserver they are Debian and Ubuntu - When I look at the catalog that gets compiled I can see that all my other resources seem to be correct, we have some osfamily/system code that checks the os and sets up apt/yum repositories. So I know that's working correct. - When I r10k my entire environment to the nodes and run it locally with 'puppet apply' there are no problems at all. - Sometimes when I clear the environment cache, the problem will go away for the first puppet run, but all future puppet runs have the same problem. - I've checked hiera, the only paramater for nrpe is to specify our ntp server I'm running puppet 4.5.1, puppetsever 2.4.0 and puppetdb 4.1.0, we use Foreman to classify our nodes with no smart paramaters etc. The only classification it does is assign a role to a node. My only ideas are: - There's some some sort of catalog/resource cache that is getting generated with the wrong nrpe resource. AFAIK this doesn't exist? As only whole catalogs are cached on the puppetserver - Somehow puppetdb might be invloved? I know puppetdb has resources/catalogs on it. Does puppetserver use puppetdb as a cache for resources/catalogs when generating a catalog for nodes? If so is there a way to debug why it would be generating a catalog with an incorrect resource for nodes? Anyone have any ideas how to debug this? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/889f44ad-1ece-4dff-a769-b91c916d4bee%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
