Passenger settings are currently - PassengerHighPerformance on PassengerUseGlobalQueue on
PassengerMaxPoolSize 3 PassengerMaxRequests 1000 PassengerPoolIdleTime 1800 I have run an strace (aka dtruss since these are on OS X clients mostly). I am seeing this in huge blocks (100s) on the nodes having issues - __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 __semwait_signal(0x50B, 0x0, 0x1) = -1 Err#60 They are also scattered throughout other sections of the run, but in smaller 4 or 5 quantities. The clients vary from 10.6 to 10.8 but the ones having issues are consistent with this error. They are also punctuated with blocks like this every once in a while - select(0x7, 0x7FFF62AECE10, 0x7FFF62AECD90, 0x7FFF62AECD10, 0x7FFF62AECD00) = 1 0 connect(0x5, 0x7F84D40FA5A0, 0x10) = -1 Err#56 fcntl(0x5, 0x4, 0x2) = 0 0 fcntl_nocancel(0x5, 0x3, 0x0) = 2 0 fcntl_nocancel(0x5, 0x3, 0x0) = 2 0 madvise(0x105128000, 0x5C000, 0x9) = 0 0 fcntl(0x5, 0x3, 0x0) = 2 0 fcntl(0x5, 0x4, 0x6) = 0 0 write(0x5, "\200p\001\003\001\0", 0x72) = 114 0 dtrace: error on enabled probe ID 1791 (ID 116: syscall::read:return): out of scratch space in action #12 at DIF offset 44 But that looks like dtrace having issues and nothing to do with Puppet. The disk on the machines check out clean. DNS is also good and consistent with our working nodes. These machines are imaged from a standard so having a one off is typically very rare, we also use Puppet to help keep things nice and tidy. I am not currently seeing network errors on the switches. This one is quite odd, thanks to everyone who has suggested things. Very much appreciated. Rob On Tuesday, February 26, 2013 11:32:33 AM UTC-7, r.yeo wrote: > > Running the agent in debug mode on a some clients and I am seeing long run > times such as this, but only after the "using pson" lines : > > Tue Feb 26 10:57:38 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 10:58:53 2013: Info: > /Stage[main]/Vpn::Config/File[/opt/cisco/vpn/.anyconnect_global]: Evaluated > in 75.03 seconds > > Tue Feb 26 10:58:53 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 11:00:08 2013: Info: > /Stage[main]/Amadeuspro/File[/etc/puppet/tmp/com.HairerSoft.AmadeusPro.plist]: > > Evaluated in 74.78 seconds > Tue Feb 26 11:00:08 2013: Info: > /Stage[main]/Adobe/File[/etc/puppet/tmp/com.adobe.Reader.plist]: Starting > to evaluate the resource > Tue Feb 26 11:00:08 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 11:01:23 2013: Info: > /Stage[main]/Adobe/File[/etc/puppet/tmp/com.adobe.Reader.plist]: Evaluated > in 75.00 seconds > > Tue Feb 26 11:01:23 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 11:02:38 2013: Info: /Stage[main]/Puppet::Snow/File[/etc/hooks/ > cleanpup.pl]: Evaluated in 75.01 seconds > > This catalog took 921.02 to complete : > > Tue Feb 26 11:02:38 2013: Notice: Finished catalog run in 916.78 seconds > Tue Feb 26 11:02:38 2013: Changes: > Tue Feb 26 11:02:38 2013: Events: > Tue Feb 26 11:02:38 2013: Resources: > Tue Feb 26 11:02:38 2013: Total: 55 > Tue Feb 26 11:02:38 2013: Skipped: 6 > Tue Feb 26 11:02:38 2013: Time: > Tue Feb 26 11:02:38 2013: Filebucket: 0.00 > Tue Feb 26 11:02:38 2013: Package: 0.00 > Tue Feb 26 11:02:38 2013: Ssh authorized key: 0.00 > Tue Feb 26 11:02:38 2013: User: 0.06 > Tue Feb 26 11:02:38 2013: Group: 0.11 > Tue Feb 26 11:02:38 2013: Exec: 0.59 > Tue Feb 26 11:02:38 2013: Config retrieval: 13.28 > Tue Feb 26 11:02:38 2013: Last run: 1361901758 > Tue Feb 26 11:02:38 2013: File: 906.98 > Tue Feb 26 11:02:38 2013: Total: 921.02 > Tue Feb 26 11:02:38 2013: Version: > Tue Feb 26 11:02:38 2013: Config: 1361893868 > Tue Feb 26 11:02:38 2013: Puppet: 3.1.0 > > Also, this isn't global as I have nodes getting identical complete configs > in 7 seconds and run times are just fine after the using pson lines. > > Example of a fast node : > > Tue Feb 26 11:22:57 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 11:22:58 2013: Info: > /Stage[main]/Amadeuspro/File[/etc/puppet/tmp/com.HairerSoft.AmadeusPro.plist]: > > Evaluated in 0.15 seconds > Tue Feb 26 11:22:58 2013: Info: > /Stage[main]/Adobe/File[/etc/puppet/tmp/com.adobe.Reader.plist]: Starting > to evaluate the resource > Tue Feb 26 11:22:58 2013: Debug: file_metadata supports formats: > b64_zlib_yaml pson raw yaml; using pson > Tue Feb 26 11:22:58 2013: Info: > /Stage[main]/Adobe/File[/etc/puppet/tmp/com.adobe.Reader.plist]: Evaluated > in 0.07 seconds > > Puppet master and clients are 3.1 with Dashboard 1.2.22 and also using > PuppetDB. > > Any thoughts why I'd have trouble and long run times on select nodes when > pson is called? > > Thanks! > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
