Re: [Puppet Users] Re: How can I diagnose hopeless performance.
On Saturday, 21 March 2015 22:08:34 UTC, Christopher Wood wrote: On Sat, Mar 21, 2015 at 12:08:07PM -0700, Robert Inder wrote: : : This is not quite the behaviour I want. But how can I stop it? In your place I'd probably just purge stale sessions with a cron job. Any particular reason why puppet's tidy is better than find-rm? Thanks for replying. I did not set up the system, so I can't tell you for sure, but I guess it was originally done by puppet because it looked like a natural thing to do -- like exactly the kind of task that Tidy is intended for! But having now found the unexpected consequences of doing the obvious thing, doing it with cron does seem like the way to to. So I'll switch to that, and manually remove all the state.yaml files to get things to a clean state. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/cea643df-ef2e-4194-a57a-a0f3e05bc48b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: How can I diagnose hopeless performance.
On Friday, 20 March 2015 08:18:15 UTC, jamese wrote: Try running puppet with --debug and --evaltrace to see where it's taking the time. Ah! As before, the evaltrace for the empty node definition lists a handful of things being evaluated in negligible times. But that out put is bracketed by debug info. It is preceded by... Debug: Loaded state in 40.54 seconds Debug: Loaded state in 90.30 seconds And followed by Debug: Stored state in 130.67 seconds Notice: Finished catalog run in 224.24 seconds So the time appears to be going on loading and saving State. So I look. And I find the answer. Our server is running applications written in Perl, using Catalyst. Those applications track user sessions.in files created below a sessions directory. We have configured Puppet to Tidy the sessions directories, by removing files over 3 days old. Currently, we have something like 1200 files in the sessions directory tree. A few hundred sessions per day. Plausible. /var/lib/puppet/state/state.yaml has *four hundred thousand* lines related to these session files. So in its state file, Puppet is keeping a record of every session file it has ever deleted. It looks like every time it starts, it reads them all in. And every time it finishes, it writes them all back out again, even though they don't exist. If I remove the state directory, puppet processes the empty node in no time at all FFS! This is not quite the behaviour I want. But how can I stop it? -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/154122c1-d228-4de6-82db-d1df8fe1bd9c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: How can I diagnose hopeless performance.
On Sat, Mar 21, 2015 at 12:08:07PM -0700, Robert Inder wrote: On Friday, 20 March 2015 08:18:15 UTC, jamese wrote: Try running puppet with --debug and --evaltrace to see where it's taking the time. Ah! As before, the evaltrace for the empty node definition lists a handful of things being evaluated in negligible times. But that out put is bracketed by debug info. It is preceded by... Debug: Loaded state in 40.54 seconds Debug: Loaded state in 90.30 seconds And followed by Debug: Stored state in 130.67 seconds Notice: Finished catalog run in 224.24 seconds So the time appears to be going on loading and saving State. So I look. And I find the answer. Our server is running applications written in Perl, using Catalyst. Those applications track user sessions.in files created below a sessions directory. We have configured Puppet to Tidy the sessions directories, by removing files over 3 days old. Currently, we have something like 1200 files in the sessions directory tree. A few hundred sessions per day. Plausible. /var/lib/puppet/state/state.yaml has four hundred thousand lines related to these session files. So in its state file, Puppet is keeping a record of every session file it has ever deleted. It looks like every time it starts, it reads them all in. And every time it finishes, it writes them all back out again, even though they don't exist. If I remove the state directory, puppet processes the empty node in no time at all FFS! This is not quite the behaviour I want. But how can I stop it? In your place I'd probably just purge stale sessions with a cron job. Any particular reason why puppet's tidy is better than find-rm? -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to [1]puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit [2]https://groups.google.com/d/msgid/puppet-users/154122c1-d228-4de6-82db-d1df8fe1bd9c%40googlegroups.com. For more options, visit [3]https://groups.google.com/d/optout. References Visible links 1. mailto:puppet-users+unsubscr...@googlegroups.com 2. https://groups.google.com/d/msgid/puppet-users/154122c1-d228-4de6-82db-d1df8fe1bd9c%40googlegroups.com?utm_medium=emailutm_source=footer 3. https://groups.google.com/d/optout -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20150321220819.GA30516%40iniquitous.heresiarch.ca. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: How can I diagnose hopeless performance.
Try running puppet with --debug and --evaltrace to see where it's taking the time. I'd be looking at DNS as that is often the culprit for unexplained things. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5a77c1c3-69c3-4202-9d3d-65bf6770619e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: How can I diagnose hopeless performance.
On Fri, Mar 20, 2015 at 4:18 AM jamese james.eckers...@gmail.com wrote: Try running puppet with --debug and --evaltrace to see where it's taking the time. I'd be looking at DNS as that is often the culprit for unexplained things. -- A tale I tell is where Puppet was taking three or more minutes to even begin to do anything. This was on an RHEL platform and it turned out my support guys has RHN configured to look for an RPM repository that was unreachable due to network restrictions. I removed the RHN plugin to yum and Puppet began to function normally. Just throwing that out there... -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAArvnv1nfPkQpWNA-by2ozYAWNAUGznpFdoAdsSgQyk0Oh0Rfw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: How can I diagnose hopeless performance.
On Friday, March 20, 2015 at 3:18:15 AM UTC-5, jamese wrote: Try running puppet with --debug and --evaltrace to see where it's taking the time. I'd be looking at DNS as that is often the culprit for unexplained things. DNS issues can indeed cause long runtimes when requests time out, but that would not explain the full CPU utilization the OP reports. If the affected machine exhibits full CPU utilization and yet much longer runtimes during Puppet runs than other machines, it follows that either the machine is doing more work during the time of catalog application than the other machines do (but not necessarily via Puppet), or the work capacity of the affected machine's hardware is less than the others'. If the affected machine is a VM, then it may be that the specs of its (virtual) hardware have been reduced, or that it is suffering from heavy contention with one or more other VMs hosted on the same physical machine. Whether physical or virtual, it might be suffering from hardware [near-]failure such as many I/O failures/retries due to a failing disk, or many memory errors detected on a marginal ECC memory module. If the physical machine is running hot for whatever reason then it may be that CPU speed has been throttled back. The information obtained from running puppet with --evaltrace might shed some light on this, as might the system log. If the problem cannot be traced back to hardware, then the question becomes what the affected machine is doing that the others are not doing. If 'top' and 'puppet --debug apply' does not illuminate this, however, then perhaps the easiest solution would be to just rebuild the machine. Since it is currently a standby anyway, that should be viable. (And since it is Puppet-managed, that should be easy, too :-)). John -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7bea7ecd-45c5-425f-912a-560726e43e1a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
