On Thu, Sep 28, 2017 at 10:42 AM, Mark Waite <mark.earl.wa...@gmail.com> wrote: > Do we have any way of associating historical acceptance test harness > failures to a cause of those failures?
Not that I am aware of. > could such data be gathered from some sample of recent runs of the > acceptance test harness, so that we know which of the tests have been most > helpful in the recent past? Maybe. I doubt things are in good enough condition to do that kind of analysis. AFAIK none of the CI jobs running the ATH have ever had a single stable run, so there is not really a baseline. > Alternately, could we add a layering concept to the acceptance test harness? > There could be "precious" tests which run every time and there could be > other tests which are part of a collection from which a few tests are > selected randomly for execution every time. Yes this is possible. > is there a way to make the acceptance test harness run > in a massively parallel fashion Yes. Does not help with the flakiness, and still wastes a tremendous amount of cloud machine time. > As a safety check of that concept, did any of the current acceptance tests > detect the regression when run with Jenkins 2.80 (or Jenkins 2.80 RC)? Yes. > Is there a JenkinsRule test which could reasonably be written to test for > the conditions that caused the bug in Jenkins 2.80? Not really; that particular issue was unusual, since it touched on the setup wizard UI which is normally suppressed by test harnesses. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0LJ6z%3DTiQe8Rt9P_D7RpZDEHt3XAWQxoq%3Dd0dVAD%2BG%3Dw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
