This is great info, but how big of a pool of ssh slaves does this become a problem at? We have 12. (And again, the problem goes away by downgrading the library.)
-- Dean From: Stephen Connolly <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, July 29, 2014 at 11:42 PM To: "[email protected]" <[email protected]> Subject: Re: SSH slave performance degradation > In my scalability testing I have found you cannot scale out ssh slaves with > /dev/random as the entropy source. You need to use /dev/./urandom (JVM bug > requires that name btw) > > The master on windows is a different story though > > On Wednesday, 30 July 2014, Mark Waite <[email protected]> wrote: >> I thought that a common default on Linux was to block if /dev/random was to >> block if the pool of random data was emptied. Refer to >> http://en.wikipedia.org/?title=/dev/random for a description. >> >> I thought that /dev/urandom did not block if the pool of random data was >> emptied. That same article describes the differences between the two. >> >> I've seen cases with some versions of Java and some Linux variants where Java >> performance suffered badly when I had emptied the pool of random data. I >> think that is why Stephen recommends using /dev/urandom so that your program >> won't block while waiting for random data. >> >> Mark Waite >> >> >> On Tue, Jul 29, 2014 at 10:44 PM, Dean Yu <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');> > wrote: >>> Obviously, going from 1.509.4 to 1.554.3 is a pretty big jump that included >>> lots and lots of changes. However, the fact that the singular act of >>> downgrading that library got us back to our prior build times is a big >>> smoking gun to me. >>> >>>> > I wonder if something changed upstream... >>> >>> From the upstream release notes: >>>> >>>> build217, 2013-06-03: >>>> >>>> - Support for SSH agent based authentication. >>>> >>>> build216, 2013-03-04: >>>> >>>> - Support of unencrypted entries in the known_hosts file. >>>> - Improved timeout handling. >>>> > BTW you are using /dev/./urandom as an entropy source for the JVM? >>> >>> Nope. Should we? >>> >>> -- Dean >>> >>> >>> From: Stephen Connolly <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> > >>> Reply-To: "[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> " >>> <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> > >>> Date: Tuesday, July 29, 2014 at 2:16 PM >>> To: "[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> " >>> <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> > >>> Subject: Re: SSH slave performance degradation >>> >>>> * KK's changes to window sizes should have *increased* performance >>>> * My connection bug fixes were surgical IIRC >>>> * Nicolas's merge of upstream seems to include an EOL change, so hard to >>>> see what changed there with the Github diff tool: >>>> https://github.com/jenkinsci/trilead-ssh2/compare/trilead-ssh2-build214-jen >>>> kins-3...trilead-ssh2-build217-jenkins-5 >>>> >>>> I wonder if something changed upstream... >>>> >>>> BTW you are using /dev/./urandom as an entropy source for the JVM? >>>> >>>> >>>> On 29 July 2014 19:51, Dean Yu <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> > wrote: >>>>> Hi folks, >>>>> We just upgraded our cluster from 1.509.4 to 1.554.3, and discovered a >>>>> significant increase in our build times. Builds that typically took ~50 to >>>>> complete started taking ~90 minutes to finish, sometimes spiking to 2 >>>>> hours. While researching, we found this JIRA[1] which reported that >>>>> downgrading the trilead-ssh2 jar solved the performance issues. >>>>> While this ticket talks specifically artifact downloads, we see that our >>>>> builds as a whole were slower. >>>>> The trilead-ssh2 dependency version was updated by [2], so it was >>>>> introduced into 1.536, show would only have made it to LTS with 1.554.1 in >>>>> April. >>>>> Looking at the trilead-ssh2 repo[3], it looks like there were a small >>>>> set of changes: >>>>> - changes by ndeloof to merge a newer upstream (build214 to build217) >>>>> - changes by stephenc to fix connection bugs >>>>> - changes by kohsuke to support package window sizes >>>>> >>>>> Anyone have thoughts on the likely culprit? Given the severity of the >>>>> performance hit we took, I'm surprised that more people haven't reported >>>>> this. >>>>> >>>>> -- Dean >>>>> >>>>> [1] https://issues.jenkins-ci.org/browse/JENKINS-20550 >>>>> [2] >>>>> https://github.com/jenkinsci/jenkins/commit/bb265c5e95b0fe39128720b9039142 >>>>> 36962db41b >>>>> [3] https://github.com/jenkinsci/trilead-ssh2/commits/master >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Jenkins Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected] >>>>> ');> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected]' >>>> );> . >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected]') >>> ;> . >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Thanks! >> Mark Waite >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected]'); >> > . >> For more options, visit https://groups.google.com/d/optout. > > > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
