> Oh and yes that crazy path is the only way to get it to work... it's a bug/feature of the JVM
I'm taking that by the lack of mention of JVM version that this is an outstanding issue? From: Stephen Connolly <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, July 30, 2014 at 7:51 AM To: "[email protected]" <[email protected]> Subject: Re: SSH slave performance degradation > on all linux machines you can just add > `-Djava.security.egd=file:/dev/./urandom` to the JVM startup command. > > This is more critical on the Jenkins master than the slaves as the slaves > typically only have one connection back to the master where as the master has > multiple slaves. > > If you have multiple slaves sharing the same machine then you would probably > need it for the slaves also. > > Windows machines do not have this issue as far as I am aware. > > Oh and yes that crazy path is the only way to get it to work... it's a > bug/feature of the JVM > > > On 30 July 2014 15:23, Mike Chmielewski <[email protected]> wrote: >> What's the most straightforward way to add this to my installation, add to >> container args for master, and the node configuration for slaves? Is this >> just needed on master or just needed on slaves? >> >> >> ----- Original message ----- >> From: Stephen Connolly <[email protected]> >> To: "[email protected]" <[email protected]> >> Subject: Re: SSH slave performance degradation >> Date: Wed, 30 Jul 2014 15:04:30 +0100 >> >> On an AWS m3.large I could not even get to 10 SSH slaves connected without >> switching to /dev/./urandom >> >> >> On 30 July 2014 14:48, Dean Yu <[email protected]> wrote: >>> This is great info, but how big of a pool of ssh slaves does this become a >>> problem at? We have 12. (And again, the problem goes away by downgrading the >>> library.) >>> >>> -- Dean >>> >>> From: Stephen Connolly <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Tuesday, July 29, 2014 at 11:42 PM >>> >>> To: "[email protected]" <[email protected]> >>> Subject: Re: SSH slave performance degradation >>> >>>> In my scalability testing I have found you cannot scale out ssh slaves with >>>> /dev/random as the entropy source. You need to use /dev/./urandom (JVM bug >>>> requires that name btw) >>>> >>>> The master on windows is a different story though >>>> >>>> On Wednesday, 30 July 2014, Mark Waite <[email protected]> wrote: >>>>> I thought that a common default on Linux was to block if /dev/random was >>>>> to block if the pool of random data was emptied. Refer to >>>>> http://en.wikipedia.org/?title=/dev/random for a description. >>>>> >>>>> I thought that /dev/urandom did not block if the pool of random data was >>>>> emptied. That same article describes the differences between the two. >>>>> >>>>> I've seen cases with some versions of Java and some Linux variants where >>>>> Java performance suffered badly when I had emptied the pool of random >>>>> data. I think that is why Stephen recommends using /dev/urandom so that >>>>> your program won't block while waiting for random data. >>>>> >>>>> Mark Waite >>>>> >>>>> >>>>> On Tue, Jul 29, 2014 at 10:44 PM, Dean Yu <[email protected]> wrote: >>>>>> Obviously, going from 1.509.4 to 1.554.3 is a pretty big jump that >>>>>> included lots and lots of changes. However, the fact that the singular >>>>>> act of downgrading that library got us back to our prior build times is a >>>>>> big smoking gun to me. >>>>>> >>>>>>> > I wonder if something changed upstream... >>>>>> >>>>>> From the upstream release notes: >>>>>>> >>>>>>> build217, 2013-06-03: >>>>>>> >>>>>>> - Support for SSH agent based authentication. >>>>>>> >>>>>>> build216, 2013-03-04: >>>>>>> >>>>>>> - Support of unencrypted entries in the known_hosts file. >>>>>>> - Improved timeout handling. >>>>>>> > BTW you are using /dev/./urandom as an entropy source for the JVM? >>>>>> >>>>>> Nope. Should we? >>>>>> >>>>>> -- Dean >>>>>> >>>>>> >>>>>> From: Stephen Connolly <[email protected]> >>>>>> Reply-To: "[email protected]" >>>>>> <[email protected]> >>>>>> Date: Tuesday, July 29, 2014 at 2:16 PM >>>>>> To: "[email protected]" <[email protected]> >>>>>> Subject: Re: SSH slave performance degradation >>>>>> >>>>>>> * KK's changes to window sizes should have *increased* performance >>>>>>> * My connection bug fixes were surgical IIRC >>>>>>> * Nicolas's merge of upstream seems to include an EOL change, so hard to >>>>>>> see what changed there with the Github diff tool: >>>>>>> https://github.com/jenkinsci/trilead-ssh2/compare/trilead-ssh2-build214- >>>>>>> jenkins-3...trilead-ssh2-build217-jenkins-5 >>>>>>> >>>>>>> I wonder if something changed upstream... >>>>>>> >>>>>>> BTW you are using /dev/./urandom as an entropy source for the JVM? >>>>>>> >>>>>>> >>>>>>> On 29 July 2014 19:51, Dean Yu <[email protected]> wrote: >>>>>>> Hi folks, >>>>>>> We just upgraded our cluster from 1.509.4 to 1.554.3, and discovered a >>>>>>> significant increase in our build times. Builds that typically took ~50 >>>>>>> to complete started taking ~90 minutes to finish, sometimes spiking to 2 >>>>>>> hours. While researching, we found this JIRA[1] which reported that >>>>>>> downgrading the trilead-ssh2 jar solved the performance issues. >>>>>>> While this ticket talks specifically artifact downloads, we see that >>>>>>> our builds as a whole were slower. >>>>>>> The trilead-ssh2 dependency version was updated by [2], so it was >>>>>>> introduced into 1.536, show would only have made it to LTS with 1.554.1 >>>>>>> in April. >>>>>>> Looking at the trilead-ssh2 repo[3], it looks like there were a small >>>>>>> set of changes: >>>>>>> - changes by ndeloof to merge a newer upstream (build214 to build217) >>>>>>> - changes by stephenc to fix connection bugs >>>>>>> - changes by kohsuke to support package window sizes >>>>>>> >>>>>>> Anyone have thoughts on the likely culprit? Given the severity of the >>>>>>> performance hit we took, I'm surprised that more people haven't reported >>>>>>> this. >>>>>>> >>>>>>> -- Dean >>>>>>> >>>>>>> [1] https://issues.jenkins-ci.org/browse/JENKINS-20550 >>>>>>> [2] >>>>>>> https://github.com/jenkinsci/jenkins/commit/bb265c5e95b0fe39128720b90391 >>>>>>> 4236962db41b >>>>>>> [3] https://github.com/jenkinsci/trilead-ssh2/commits/master >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Jenkins Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Jenkins Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "Jenkins Developers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>>> email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks! >>>>> Mark Waite >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Jenkins Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> Sent from my phone >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> Mike Chmielewski >> [email protected] >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
