Hi, You do not necessarily need very large setups to do these performance tests. What you need to be doing is performing proper measurements on the right things. I would even go so far as to say that large setups might actually make things difficult to disentangle.
Since it sounds like the project has got nothing in terms of systematic performance measurements, I would advise to start smaller. You certainly want larger setups to be part of the mix, but I think the first focus should be on the smaller setups for a baseline. Also, one should check behaviour when you have a large mix of plugins. Again, the more important point is measurement. To anyone who is setting this up who is not a physicist by training (maybe you need reminding if you are): a single number is not a measurement. At a minimum, a measurement is 2 numbers, lower and upper range. And even better is a mean + standard deviation or confidence interval. Why am I pointing this out you may ask. Well, if I supposedly measured the time for something and tell you it took 30 seconds. Then I change some code and time again and get 25 seconds. Have I improved things? You would maybe say yes. But what if I tell you the measurement was 30 +/- 10 versus 25 +/- 10 seconds? I haven't really improved anything, now have I. It's just noise. So, if we are serious about test driven software development, we should also be serious about measurement. It is also important to record and keep trends of the timings. There will be outliers and there will be weird stuff in the trends from time to time, which needs to be checked, analysed and understood. As a last point on measurement, I dont know if there is an easy way to get profile information, but a break down of how much CPU and I/O each plugin or core service consumes should be a goal. If you can measure that, you can make quick progress on weeding out the culprits. Kind regards. Artur ________________________________ From: [email protected] [[email protected]] on behalf of Michael Neale [[email protected]] Sent: 02 October 2015 02:18 To: Jenkins Developers Subject: Re: performance labs Oh wow - they may be a perfect test workload. Do you know if boot up times are in the many many minutes for those instances? Some data on the jenkins_home dir sizes? It would be ideal to use opensource workloads (even if it is a point in time) vs something contrived, or a scrubbed version of a private users data that has donated it, however it would want to be pretty hefty (not necessarily 2TB jenkins_homes that I have heard of, or 40 minute boot up, but something up there would be nice). I guess the next step is an initial scope of what we want to measure. To keep things focussed I am thinking or boot up to job load time, and listing a few things. On Thursday, October 1, 2015 at 5:55:08 PM UTC+10, Andrew Bayer wrote: ...and I can most likely provide builds.apache.org<http://builds.apache.org>'s jobs/builds/load/etc as a use case. A. On Thu, Oct 1, 2015 at 9:54 AM, Andrew Bayer <[email protected]<UrlBlockedError.aspx>> wrote: +1 - that'd be fantastic. I'd love to help with that. A. On Thu, Oct 1, 2015 at 4:50 AM, Michael Neale <[email protected]<UrlBlockedError.aspx>> wrote: Hey all - I have thought it would be a great idea to have some quasi formal "performance lab" setups for Jenkins. Recently around Jenkins 2.0 planning threads there have been lots of comments around performance challenges. Often things like launch time (talking many minutes to an hour for large workspaces - launch times are probably a good proxy for a whole lot of issues, but there are other issues too). At JUC west there was an excellent talk by Akshay Dayal from Google, on scaling jenkins. I highly recommend flicking through the slides<https://www.cloudbees.com/jenkins/juc-2015/abstracts/us-west/02-01-1600> or watching the talk<https://www.youtube.com/watch?v=9-DUVroz7yk&index=16&list=PLvBBnHmZuNQKyjKInevHYsRq8J7Q1I6Fq> if you have time. Basically, they had some performance goals and started by setting up measurements and test scenarios to validate their progress - both around scalability of slaves (an interesting issue) but also on bootup time (time to recovery) which is very interesting. It reminded me that to improve something like this you kind of need easily repeatable measurements in controlled environments, which currently I don't think the Jenkins project has set up? (correct me if wrong). I know Stephen Connolly did some work a few years back on slave scalability which was interesting (building out a test suite infrastructure), but I am not aware of subsequent efforts. Is this something people would be interested in? Having either large sample JENKINS_HOME specimens or test code that can generate pathological data would be required, as well as automation around running it on a variety of machines (not necessarily cloud, ideally want to be testing code not cloud infrastructure). -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<UrlBlockedError.aspx>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/76a12929-8f10-4b50-bf01-04cc77768149%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-dev/76a12929-8f10-4b50-bf01-04cc77768149%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/297b9981-edc5-49e2-b46f-f0b49183f32c%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-dev/297b9981-edc5-49e2-b46f-f0b49183f32c%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3C350C9D1C9DEE47BAEA43102B433C9EBE5DDC24%40HQMBX2.ads.eso.org. For more options, visit https://groups.google.com/d/optout.
