On Wed, 2010-03-10 at 17:18 -0800, Douglas Garstang wrote: > On Wed, Mar 10, 2010 at 1:38 PM, Douglas Garstang > <[email protected]> wrote: > > On Wed, Mar 10, 2010 at 1:34 PM, Douglas Garstang > > <[email protected]> wrote: > >> On Wed, Mar 10, 2010 at 1:17 PM, Brice Figureau > >> <[email protected]> wrote: > >>> On 10/03/10 22:06, Douglas Garstang wrote: > >>>> So, it became apparent to me, after emailing someone off list, that > >>>> managing a lot of files in deep directory structures might be part of > >>>> the cause. > >>>> > >>>> We are running 10 instances of JBOSS and 10 instances of tomcat on > >>>> each of these servers. Don't ask me why, it's just the way it was done > >>>> before I arrived and changing it is not trivial. > >>>> > >>>> On disk, each instance of JBOSS starts at > >>>> /opt/jboss/current/server/tfelN (where N is the instance number) > >>>> > >>>> and each instance of tomcat starts at: > >>>> /opt/tomcat/tfelN/starterkit/current (where N is the instance number) > >>> > >>> Do you source the whole hierarchy? > >>> Or do you only manage it? > >>> > >>>> I manually looked through the puppet config and counted 25 unique > >>>> files that are being managed for jboss and tomcat within these paths. > >>>> If you do the math, 25 x 10 x 2 = 500. That's therefore (currently) > >>>> 500 unique files that are being managed in these deep directory > >>>> structures. Could that potentially be the reason behind puppets crap > >>>> performance? > >>> > >>> What do you manage for those files? > >>> But no, 500 doesn't seem like a high number to me. > >>> > >>> You mentioned in another e-mail in this thread that the problem is more > >>> the 20 minutes run than the CPU. > >>> Could it be possible you have many "slow" execs? > >>> Or you manage many packages? > >>> > >>> This also reminds me Ohad's bug: > >>> http://projects.reductivelabs.com/issues/1719 > >>> > >>> At this stage you should probably run puppetd on the console in --debug > >>> to see what happens (and run with --summarize too) and if it stalls. > >> > >> I just ran puppet in debug mode and it was obvious that most of the > >> puppet run time was spent in checksumming files. > >> > >> Eg: > >> > >> debug: > >> //Node[app01.fr.xxx.com]/Jboss::Instance[tfel8]/File[/opt/jboss/current/server/tfel8/conf/jboss.web/localhost/rewrite.properties]: > >> Creating checksum {md5}f5d16bcc20b92631eb59514018fd34e5 > >> > >> ... takes a long time to run. Multiple that by several hundred files...
That's particularly strange. Note that when puppet displays this, the checksum has already been computed. So if you see a "downtime" after it prints a checksum change and before the next, it is well possible Puppet is sourcing the file from the master. In which case the bottleneck would be the fileserving and not the checksumming. Note again, if the file is not sourced, nor a template you don't need to checksum it. You need to give us more information about those files (ie size, sourced/template?...) Can you run with --summarize, puppet will display in what operation it is taking time. > >> However, when I run this on the command line: > >> md5sum > >> /opt/jboss/current/server/tfel8/conf/jboss.web/localhost/rewrite.properties > >> > >> ... the result is instananeous... So... is puppet using a ruby library > >> for performing md5 checksums? Is that where the performance bottle > >> neck could be? I don't know. > >> Doug > > Jeez..... it went quiet in this thread didn't it... I'm not in your timezone if that's what you were asking. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
