If broadening scope a bit, I would like to include memory footprint measurements too (something I spend time thinking about). Stability of of more pressing importance I agree.
On Thu, 15 Oct 2015 at 12:04 AM, Andrew Bayer <[email protected]> wrote: > Yeah, stability is my biggest concern, and an even harder thing to test > for than performance. Might be worth scraping through JIRA to find examples > of behavior that tends to trigger instability in various ways to come up > with some ideas... > > A. > > On Wed, Oct 14, 2015 at 1:36 PM, Artur Szostak <[email protected]> > wrote: > >> The thread has been focusing on performance in terms of speed. But let me >> add another performance dimension that honestly is much more important to >> me right now (and causing me a lot of plain): >> performance as in stability. >> >> The following kinds of tests might go a long way in first quantitatively >> evaluating how stable Jenkins is and fixing these problems down the line. >> - Perform continual start/stop cycles of the Jenkins master under various >> loads (system stress). >> - Perform continual build slave start/build/stop cycles under various >> loads of the system and network. Ideally one would add simulations of >> intermittent network failure and check that Jenkins follows the expected >> error path. >> >> I dont know about other people's experience, but I see that above a >> handful of build slave nodes one starts seeing a lot of connectivity and >> start up / shutdown issues. I also suspect there are a number of race >> conditions in there. >> >> ------------------------------ >> *From:* [email protected] [[email protected]] >> on behalf of Michael Neale [[email protected]] >> *Sent:* 14 October 2015 03:52 >> *To:* Jenkins Dev >> *Subject:* Re: performance labs >> >> Ok so it sounds like exhuming Stephens scalability stuff (not sure if it >> did startup time, but it doesn't sound like it would be hard to ad) would >> be a great place to start. Like you said, turning the dials and seeing what >> happens is super useful. Even on VMs (vs bare metal) would be informative >> as its increasingly common to run Jenkins not on bare hardware. >> On Wed, 14 Oct 2015 at 2:42 AM, Jesse Glick <[email protected]> wrote: >> >>> On Tue, Oct 13, 2015 at 7:29 AM, Michael Neale <[email protected]> >>> wrote: >>> > Looking at the variation of times people see, I am questioning the >>> utility >>> > of a generic test suite. Things vary so much there may be too many >>> variables >>> > at play to make something like this useful right now. >>> >>> Well a generic test suite is not going to predict any given >>> installation’s performance, of course. But it can serve a controlled >>> baseline by which you can measure the effects of changes. And many >>> widely applicable bugs, like the ones Google engineers found, can be >>> reproduced this way. When Stephen and I were poring over results from >>> sample tests using his scalability framework, which did really generic >>> stuff—run lots of builds from lots of jobs, each build producing gobs >>> of output—it was immediately clear what was broken. You set n=10 and >>> all is well. You set n=500 and things start to look worse. You set >>> n=10000 and the system basically hangs, and you look at a thread dump, >>> and oh yes a thousand threads are waiting on this one lock for no good >>> reason. So you fix that problem and rerun the test and you find the >>> next problem. >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Jenkins Developers" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/jenkinsci-dev/1F9DHyMOutw/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1kkepm0P%2B9QzLzZq9a5dmjzos3jYu6r4W6r4wbvXPX2A%40mail.gmail.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi5sSUG%3DY9rNcQ4MoZLdV1%3DWq9uQoYt6rGTrB8LJOL1poQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi5sSUG%3DY9rNcQ4MoZLdV1%3DWq9uQoYt6rGTrB8LJOL1poQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/3C350C9D1C9DEE47BAEA43102B433C9EBE5DFDE5%40HQMBX2.ads.eso.org >> <https://groups.google.com/d/msgid/jenkinsci-dev/3C350C9D1C9DEE47BAEA43102B433C9EBE5DFDE5%40HQMBX2.ads.eso.org?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Developers" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-dev/1F9DHyMOutw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdObsVaKYX4teixAeC3NiRnOknNm-NGCW-74vERmC9%2Bpi0Q%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdObsVaKYX4teixAeC3NiRnOknNm-NGCW-74vERmC9%2Bpi0Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi6SAgAPHgNNM9PT%2B_aRyG49Njt%3Df3eM33ruHcxiiC_cmQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
