What's the most straightforward way to add this to my installation, add to container args for master, and the node configuration for slaves? Is this just needed on master or just needed on slaves?
----- Original message ----- From: Stephen Connolly <[1][email protected]> To: "[2][email protected]" <[3][email protected]> Subject: Re: SSH slave performance degradation Date: Wed, 30 Jul 2014 15:04:30 +0100 On an AWS m3.large I could not even get to 10 SSH slaves connected without switching to /dev/./urandom On 30 July 2014 14:48, Dean Yu <[4][email protected]> wrote: This is great info, but how big of a pool of ssh slaves does this become a problem at? We have 12. (And again, the problem goes away by downgrading the library.) -- Dean From: Stephen Connolly <[5][email protected]> Reply-To: "[6][email protected]" <[7][email protected]> Date: Tuesday, July 29, 2014 at 11:42 PM To: "[8][email protected]" <[9][email protected]> Subject: Re: SSH slave performance degradation In my scalability testing I have found you cannot scale out ssh slaves with /dev/random as the entropy source. You need to use /dev/./urandom (JVM bug requires that name btw) The master on windows is a different story though On Wednesday, 30 July 2014, Mark Waite <[10][email protected]> wrote: I thought that a common default on Linux was to block if /dev/random was to block if the pool of random data was emptied. Refer to [11]http://en.wikipedia.org/?title=/dev/random for a description. I thought that /dev/urandom did not block if the pool of random data was emptied. That same article describes the differences between the two. I've seen cases with some versions of Java and some Linux variants where Java performance suffered badly when I had emptied the pool of random data. I think that is why Stephen recommends using /dev/urandom so that your program won't block while waiting for random data. Mark Waite On Tue, Jul 29, 2014 at 10:44 PM, Dean Yu <[email protected]> wrote: Obviously, going from 1.509.4 to 1.554.3 is a pretty big jump that included lots and lots of changes. However, the fact that the singular act of downgrading that library got us back to our prior build times is a big smoking gun to me. > I wonder if something changed upstream... >From the upstream release notes: build217, 2013-06-03: - Support for SSH agent based authentication. build216, 2013-03-04: - Support of unencrypted entries in the known_hosts file. - Improved timeout handling. > BTW you are using /dev/./urandom as an entropy source for the JVM? Nope. Should we? -- Dean From: Stephen Connolly <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, July 29, 2014 at 2:16 PM To: "[email protected]" <[email protected]> Subject: Re: SSH slave performance degradation * KK's changes to window sizes should have *increased* performance * My connection bug fixes were surgical IIRC * Nicolas's merge of upstream seems to include an EOL change, so hard to see what changed there with the Github diff tool: [12]https://github.com/jenkinsci/trilead-ssh2/compare/tri lead-ssh2-build214-jenkins-3...trilead-ssh2-build217-jenkins-5 I wonder if something changed upstream... BTW you are using /dev/./urandom as an entropy source for the JVM? On 29 July 2014 19:51, Dean Yu <[email protected]> wrote: Hi folks, We just upgraded our cluster from 1.509.4 to 1.554.3, and discovered a significant increase in our build times. Builds that typically took ~50 to complete started taking ~90 minutes to finish, sometimes spiking to 2 hours. While researching, we found this JIRA[1] which reported that downgrading the trilead-ssh2 jar solved the performance issues. While this ticket talks specifically artifact downloads, we see that our builds as a whole were slower. The trilead-ssh2 dependency version was updated by [2], so it was introduced into 1.536, show would only have made it to LTS with 1.554.1 in April. Looking at the trilead-ssh2 repo[3], it looks like there were a small set of changes: - changes by ndeloof to merge a newer upstream (build214 to build217) - changes by stephenc to fix connection bugs - changes by kohsuke to support package window sizes Anyone have thoughts on the likely culprit? Given the severity of the performance hit we took, I'm surprised that more people haven't reported this. -- Dean [1] [13]https://issues.jenkins-ci.org/browse/JENKINS-20550 [2] [14]https://github.com/jenkinsci/jenkins/commit/bb265c5e95b 0fe39128720b903914236962db41b [3] [15]https://github.com/jenkinsci/trilead-ssh2/commits/maste r -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit [16]https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit [17]https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit [18]https://groups.google.com/d/optout. -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit [19]https://groups.google.com/d/optout. -- Sent from my phone -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [20][email protected]. For more options, visit [21]https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [22][email protected]. For more options, visit [23]https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [24][email protected]. For more options, visit [25]https://groups.google.com/d/optout. -- Mike Chmielewski [email protected] References 1. mailto:[email protected] 2. mailto:[email protected] 3. mailto:[email protected] 4. mailto:[email protected] 5. mailto:[email protected] 6. mailto:[email protected] 7. mailto:[email protected] 8. mailto:[email protected] 9. mailto:[email protected] 10. mailto:[email protected] 11. http://en.wikipedia.org/?title=/dev/random 12. https://github.com/jenkinsci/trilead-ssh2/compare/trilead-ssh2-build214-jenkins-3...trilead-ssh2-build217-jenkins-5 13. https://issues.jenkins-ci.org/browse/JENKINS-20550 14. https://github.com/jenkinsci/jenkins/commit/bb265c5e95b0fe39128720b903914236962db41b 15. https://github.com/jenkinsci/trilead-ssh2/commits/master 16. https://groups.google.com/d/optout 17. https://groups.google.com/d/optout 18. https://groups.google.com/d/optout 19. https://groups.google.com/d/optout 20. mailto:[email protected] 21. https://groups.google.com/d/optout 22. mailto:[email protected] 23. https://groups.google.com/d/optout 24. mailto:[email protected] 25. https://groups.google.com/d/optout -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
