The thread has been focusing on performance in terms of speed. But let me add another performance dimension that honestly is much more important to me right now (and causing me a lot of plain): performance as in stability.
The following kinds of tests might go a long way in first quantitatively evaluating how stable Jenkins is and fixing these problems down the line. - Perform continual start/stop cycles of the Jenkins master under various loads (system stress). - Perform continual build slave start/build/stop cycles under various loads of the system and network. Ideally one would add simulations of intermittent network failure and check that Jenkins follows the expected error path. I dont know about other people's experience, but I see that above a handful of build slave nodes one starts seeing a lot of connectivity and start up / shutdown issues. I also suspect there are a number of race conditions in there. ________________________________ From: [email protected] [[email protected]] on behalf of Michael Neale [[email protected]] Sent: 14 October 2015 03:52 To: Jenkins Dev Subject: Re: performance labs Ok so it sounds like exhuming Stephens scalability stuff (not sure if it did startup time, but it doesn't sound like it would be hard to ad) would be a great place to start. Like you said, turning the dials and seeing what happens is super useful. Even on VMs (vs bare metal) would be informative as its increasingly common to run Jenkins not on bare hardware. On Wed, 14 Oct 2015 at 2:42 AM, Jesse Glick <[email protected]<mailto:[email protected]>> wrote: On Tue, Oct 13, 2015 at 7:29 AM, Michael Neale <[email protected]<mailto:[email protected]>> wrote: > Looking at the variation of times people see, I am questioning the utility > of a generic test suite. Things vary so much there may be too many variables > at play to make something like this useful right now. Well a generic test suite is not going to predict any given installation’s performance, of course. But it can serve a controlled baseline by which you can measure the effects of changes. And many widely applicable bugs, like the ones Google engineers found, can be reproduced this way. When Stephen and I were poring over results from sample tests using his scalability framework, which did really generic stuff—run lots of builds from lots of jobs, each build producing gobs of output—it was immediately clear what was broken. You set n=10 and all is well. You set n=500 and things start to look worse. You set n=10000 and the system basically hangs, and you look at a thread dump, and oh yes a thousand threads are waiting on this one lock for no good reason. So you fix that problem and rerun the test and you find the next problem. -- You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/1F9DHyMOutw/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]<mailto:jenkinsci-dev%[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1kkepm0P%2B9QzLzZq9a5dmjzos3jYu6r4W6r4wbvXPX2A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi5sSUG%3DY9rNcQ4MoZLdV1%3DWq9uQoYt6rGTrB8LJOL1poQ%40mail.gmail.com<https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi5sSUG%3DY9rNcQ4MoZLdV1%3DWq9uQoYt6rGTrB8LJOL1poQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3C350C9D1C9DEE47BAEA43102B433C9EBE5DFDE5%40HQMBX2.ads.eso.org. For more options, visit https://groups.google.com/d/optout.
