Agree on all counts, as long as it is sufficiently large/complicated enough to exercise enough code that it represents what people are observing.
Certainly the result would have to be a trend of some meaningful statistic over time, with permutations of versions. I think sticking to launch time (or time to launch/render some important page) is worth measuring. I guess some tool people can clone and try themselves (which implies something that generates a large-enough workspace would be great, but downloading a tar.gz of a large one could also do) would be great to encourage experimentation and ultimately profiling. Obviously to get a trend over time we need somewhere to run it regularly against versions and permutations of plugins. Quickly gets complicated. So where to start, a repo with some parametrised launch script people can try? Use Jenkins itself to test launch times and calculate and establish trends? On Friday, October 2, 2015 at 7:12:24 PM UTC+10, Artur Szostak wrote: > > Hi, > > You do not necessarily need very large setups to do these performance > tests. What you need to be doing is performing proper measurements on the > right things. I would even go so far as to say that large setups might > actually make things difficult to disentangle. > > Since it sounds like the project has got nothing in terms of systematic > performance measurements, I would advise to start smaller. You certainly > want larger setups to be part of the mix, but I think the first focus > should be on the smaller setups for a baseline. Also, one should check > behaviour when you have a large mix of plugins. > > Again, the more important point is measurement. To anyone who is setting > this up who is not a physicist by training (maybe you need reminding if you > are): a single number is not a measurement. At a minimum, a measurement is > 2 numbers, lower and upper range. And even better is a mean + standard > deviation or confidence interval. Why am I pointing this out you may ask. > Well, if I supposedly measured the time for something and tell you it took > 30 seconds. Then I change some code and time again and get 25 seconds. Have > I improved things? You would maybe say yes. But what if I tell you the > measurement was 30 +/- 10 versus 25 +/- 10 seconds? I haven't really > improved anything, now have I. It's just noise. So, if we are serious about > test driven software development, we should also be serious about > measurement. > > It is also important to record and keep trends of the timings. There will > be outliers and there will be weird stuff in the trends from time to time, > which needs to be checked, analysed and understood. > > As a last point on measurement, I dont know if there is an easy way to get > profile information, but a break down of how much CPU and I/O each plugin > or core service consumes should be a goal. If you can measure that, you can > make quick progress on weeding out the culprits. > > Kind regards. > > Artur > > > ------------------------------ > *From:* [email protected] <javascript:> [ > [email protected] <javascript:>] on behalf of Michael Neale [ > [email protected] <javascript:>] > *Sent:* 02 October 2015 02:18 > *To:* Jenkins Developers > *Subject:* Re: performance labs > > Oh wow - they may be a perfect test workload. Do you know if boot up times > are in the many many minutes for those instances? Some data on the > jenkins_home dir sizes? > It would be ideal to use opensource workloads (even if it is a point in > time) vs something contrived, or a scrubbed version of a private users data > that has donated it, however it would want to be pretty hefty (not > necessarily 2TB jenkins_homes that I have heard of, or 40 minute boot up, > but something up there would be nice). > > I guess the next step is an initial scope of what we want to measure. To > keep things focussed I am thinking or boot up to job load time, and listing > a few things. > > > > > On Thursday, October 1, 2015 at 5:55:08 PM UTC+10, Andrew Bayer wrote: >> >> ...and I can most likely provide builds.apache.org's >> jobs/builds/load/etc as a use case. >> >> A. >> >> On Thu, Oct 1, 2015 at 9:54 AM, Andrew Bayer <[email protected] >> <http://UrlBlockedError.aspx>> wrote: >> >>> +1 - that'd be fantastic. I'd love to help with that. >>> >>> A. >>> >>> On Thu, Oct 1, 2015 at 4:50 AM, Michael Neale <[email protected] >>> <http://UrlBlockedError.aspx>> wrote: >>> >>>> Hey all - I have thought it would be a great idea to have some quasi >>>> formal "performance lab" setups for Jenkins. >>>> >>>> Recently around Jenkins 2.0 planning threads there have been lots of >>>> comments around performance challenges. Often things like launch time >>>> (talking many minutes to an hour for large workspaces - launch times are >>>> probably a good proxy for a whole lot of issues, but there are other >>>> issues >>>> too). >>>> >>>> At JUC west there was an excellent talk by Akshay Dayal from Google, on >>>> scaling jenkins. I highly recommend flicking through the slides >>>> <https://www.cloudbees.com/jenkins/juc-2015/abstracts/us-west/02-01-1600> >>>> or watching the talk >>>> <https://www.youtube.com/watch?v=9-DUVroz7yk&index=16&list=PLvBBnHmZuNQKyjKInevHYsRq8J7Q1I6Fq> >>>> >>>> if you have time. >>>> >>>> Basically, they had some performance goals and started by setting up >>>> measurements and test scenarios to validate their progress - both around >>>> scalability of slaves (an interesting issue) but also on bootup time (time >>>> to recovery) which is very interesting. It reminded me that to improve >>>> something like this you kind of need easily repeatable measurements in >>>> controlled environments, which currently I don't think the Jenkins project >>>> has set up? (correct me if wrong). >>>> >>>> I know Stephen Connolly did some work a few years back on slave >>>> scalability which was interesting (building out a test suite >>>> infrastructure), but I am not aware of subsequent efforts. >>>> >>>> Is this something people would be interested in? >>>> >>>> Having either large sample JENKINS_HOME specimens or test code that can >>>> generate pathological data would be required, as well as automation around >>>> running it on a variety of machines (not necessarily cloud, ideally want >>>> to >>>> be testing code not cloud infrastructure). >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected] >>>> <http://UrlBlockedError.aspx>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-dev/76a12929-8f10-4b50-bf01-04cc77768149%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-dev/76a12929-8f10-4b50-bf01-04cc77768149%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/297b9981-edc5-49e2-b46f-f0b49183f32c%40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/297b9981-edc5-49e2-b46f-f0b49183f32c%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/41e65a0a-7d67-4f26-8aca-64623a5bd3cc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
