[Puppet Users] Re: Hiera Performance Testing
Thank you for your input Daniele! We're currently working to move to Puppet 3 but we've got such a large code base, so many people committing to the repo, and so much refactoring to do that it's a very complex and long term project. But I like a challenge! ;-) On Wednesday, April 16, 2014 1:29:41 AM UTC-6, Daniele Sluijters wrote: Ah, Then I'm afraid you're on your own. The instrumentation framework got added and expanded upon somewhere in the 3.x series, I believe 3.2 was the first to be able to give some useful information and as of 3.5 the API is actually public so that others can build on it. As far as Hiera goes, lookup times depend on a bunch of things. Part of it is the complexity of the hierarchy, the more places it might need to look the longer it will take. If you do a 'normal' hiera call it'll exit on the first match so depending on how quickly it can find a match it'll take a bit longer or not. Then there's the hiera_array and hiera_hash calls which search the whole hierarchy and build up an array or hash of all the returned values which will take longer than a hiera call. If you really want to get an indication I'd say just switch a few modules to hiera and compare compile times. You could also start adding Hiera.debug() calls with time information in Hiera's backend.rb. In all honesty though, I'd suggest you get to upgrading to 3.4+ first. That'll give you a bit of a performance boost and 3.4+ doesn't require Hiera, it just ships with it. It'll also enable a bunch of things that will make your life more pleasant as a module author or maintainer. If you're using opensource modules it's likely they'll shortly start dropping 2.7 support if they haven't already with the coming of Puppet 3.6 and 4.x. -- Daniele Sluijters On Monday, 14 April 2014 19:35:37 UTC+2, David Danzilio wrote: Unfortunately, and I probably should've mentioned this in the OP, we're currently stuck on the 2.7 series. On Monday, April 14, 2014 1:11:36 PM UTC-4, Daniele Sluijters wrote: Hi, I think a good start would be to turn on profiling, aka profiling=true in puppet.conf or puppet agent --profile. The output will show up in the master's logs. -- Daniele Sluijters On Friday, 11 April 2014 22:55:45 UTC+2, David Danzilio wrote: Hi Everybody. I'm trying to come up with a way to understand how Hiera is going to affect compile times on our Puppet masters. We've got just over 100K lines of Puppet code, and thousands of hosts across a few environments. I know there are a lot of variables to this problem, but I'm drawing a blank on a good way to test this. Has anybody run into this? Is there a simple way to do this that I'm just not seeing? Thanks! David -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4ba63067-0897-4893-8b02-adf73bd4a697%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Hiera Performance Testing
Thank you Nan! On Wednesday, April 16, 2014 10:31:13 AM UTC-6, Nan Liu wrote: On Mon, Apr 14, 2014 at 12:35 PM, David Danzilio david.d...@gmail.comjavascript: wrote: Unfortunately, and I probably should've mentioned this in the OP, we're currently stuck on the 2.7 series. You should be able to run puppet master --compile certname and time it to get a coarse measurement of before and after. Just make sure you take a good random sample from a variety of systems with different classes. HTH, Nan -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/28ada285-3247-408f-9c0f-01daf40649dd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: Hiera Performance Testing
Ah, Then I'm afraid you're on your own. The instrumentation framework got added and expanded upon somewhere in the 3.x series, I believe 3.2 was the first to be able to give some useful information and as of 3.5 the API is actually public so that others can build on it. As far as Hiera goes, lookup times depend on a bunch of things. Part of it is the complexity of the hierarchy, the more places it might need to look the longer it will take. If you do a 'normal' hiera call it'll exit on the first match so depending on how quickly it can find a match it'll take a bit longer or not. Then there's the hiera_array and hiera_hash calls which search the whole hierarchy and build up an array or hash of all the returned values which will take longer than a hiera call. If you really want to get an indication I'd say just switch a few modules to hiera and compare compile times. You could also start adding Hiera.debug() calls with time information in Hiera's backend.rb. In all honesty though, I'd suggest you get to upgrading to 3.4+ first. That'll give you a bit of a performance boost and 3.4+ doesn't require Hiera, it just ships with it. It'll also enable a bunch of things that will make your life more pleasant as a module author or maintainer. If you're using opensource modules it's likely they'll shortly start dropping 2.7 support if they haven't already with the coming of Puppet 3.6 and 4.x. -- Daniele Sluijters On Monday, 14 April 2014 19:35:37 UTC+2, David Danzilio wrote: Unfortunately, and I probably should've mentioned this in the OP, we're currently stuck on the 2.7 series. On Monday, April 14, 2014 1:11:36 PM UTC-4, Daniele Sluijters wrote: Hi, I think a good start would be to turn on profiling, aka profiling=true in puppet.conf or puppet agent --profile. The output will show up in the master's logs. -- Daniele Sluijters On Friday, 11 April 2014 22:55:45 UTC+2, David Danzilio wrote: Hi Everybody. I'm trying to come up with a way to understand how Hiera is going to affect compile times on our Puppet masters. We've got just over 100K lines of Puppet code, and thousands of hosts across a few environments. I know there are a lot of variables to this problem, but I'm drawing a blank on a good way to test this. Has anybody run into this? Is there a simple way to do this that I'm just not seeing? Thanks! David -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/2d3bf99d-6a06-485c-8f6f-c2f7731ce445%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Hiera Performance Testing
On Mon, Apr 14, 2014 at 12:35 PM, David Danzilio david.danzi...@gmail.comwrote: Unfortunately, and I probably should've mentioned this in the OP, we're currently stuck on the 2.7 series. You should be able to run puppet master --compile certname and time it to get a coarse measurement of before and after. Just make sure you take a good random sample from a variety of systems with different classes. HTH, Nan -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACqVBqC0O0As8K0ATD5%2BizdTKJ%3DKocA0hRcw5CsPhZuXr8F67Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: Hiera Performance Testing
Hi, I think a good start would be to turn on profiling, aka profiling=true in puppet.conf or puppet agent --profile. The output will show up in the master's logs. -- Daniele Sluijters On Friday, 11 April 2014 22:55:45 UTC+2, David Danzilio wrote: Hi Everybody. I'm trying to come up with a way to understand how Hiera is going to affect compile times on our Puppet masters. We've got just over 100K lines of Puppet code, and thousands of hosts across a few environments. I know there are a lot of variables to this problem, but I'm drawing a blank on a good way to test this. Has anybody run into this? Is there a simple way to do this that I'm just not seeing? Thanks! David -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4a5c072e-3298-4d2c-9e77-ed8abf812cd3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: Hiera Performance Testing
Unfortunately, and I probably should've mentioned this in the OP, we're currently stuck on the 2.7 series. On Monday, April 14, 2014 1:11:36 PM UTC-4, Daniele Sluijters wrote: Hi, I think a good start would be to turn on profiling, aka profiling=true in puppet.conf or puppet agent --profile. The output will show up in the master's logs. -- Daniele Sluijters On Friday, 11 April 2014 22:55:45 UTC+2, David Danzilio wrote: Hi Everybody. I'm trying to come up with a way to understand how Hiera is going to affect compile times on our Puppet masters. We've got just over 100K lines of Puppet code, and thousands of hosts across a few environments. I know there are a lot of variables to this problem, but I'm drawing a blank on a good way to test this. Has anybody run into this? Is there a simple way to do this that I'm just not seeing? Thanks! David -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c6f01c1e-cd5a-456c-b7d9-e9fa03a15a30%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
