On Tue, Oct 13, 2015 at 7:29 AM, Michael Neale <[email protected]> wrote: > Looking at the variation of times people see, I am questioning the utility > of a generic test suite. Things vary so much there may be too many variables > at play to make something like this useful right now.
Well a generic test suite is not going to predict any given installation’s performance, of course. But it can serve a controlled baseline by which you can measure the effects of changes. And many widely applicable bugs, like the ones Google engineers found, can be reproduced this way. When Stephen and I were poring over results from sample tests using his scalability framework, which did really generic stuff—run lots of builds from lots of jobs, each build producing gobs of output—it was immediately clear what was broken. You set n=10 and all is well. You set n=500 and things start to look worse. You set n=10000 and the system basically hangs, and you look at a thread dump, and oh yes a thousand threads are waiting on this one lock for no good reason. So you fix that problem and rerun the test and you find the next problem. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1kkepm0P%2B9QzLzZq9a5dmjzos3jYu6r4W6r4wbvXPX2A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
