On Thu 28 Sep 2017 at 19:11, Mark Waite <[email protected]> wrote:
> On Thu, Sep 28, 2017 at 11:43 AM Jesse Glick <[email protected]> wrote: > >> On Thu, Sep 28, 2017 at 10:42 AM, Mark Waite <[email protected]> >> wrote: >> > Do we have any way of associating historical acceptance test harness >> > failures to a cause of those failures? >> >> Not that I am aware of. >> >> > could such data be gathered from some sample of recent runs of the >> > acceptance test harness, so that we know which of the tests have been >> most >> > helpful in the recent past? >> >> Maybe. I doubt things are in good enough condition to do that kind of >> analysis. AFAIK none of the CI jobs running the ATH have ever had a >> single stable run, so there is not really a baseline. >> >> > Ah, so that means the current acceptance test harness is unlikely to be > trusted in its entirety by anyone. A failure in the entire acceptance test > harness is probably only used rarely to stop a release. > > I support deleting tests of questionable value. > I don’t want to knock the contributors to the ATH, but my long standing view is that writing good acceptance tests is a high skill and not something that can be easily crowdsourced. Acceptance tests are, by their nature, among the slowest tests. My long standing view of the ATH is that there is a fundamental flaw in using the same harness to drive as to verify *because* any change to that harness has the potential to invalidate any tests using the modified harness... it is all too easy to turn a good test into a test giving a false positive... I think an ATH that had two sets of tests, one driving by the filesystem / CLI and verifying via the UI while the other drives by the UI and verified by the filesystem / CLI would be much much stronger (and faster too). In most cases you wouldn’t be changing the two harnesses at the same time, so a failing test will indicate a failing test. We need an ATH that can be realistically run by humans in an hour or two... trim the test cases to get there, moving the tested functionality higher up to be tests that run faster, eg JenkinsRule or better yet pure unit tests. That’s just my view, feel free to ignore. (Because some people confuse my interaction style I am going to try and clarify) I’m bowing out unless someone specifically asks me for clarification on my recommendations or I see people miss-representing my opinion... this is normally the way I interact... just people have a real habit of dragging me back in and that makes me look like someone who rabbits on and on. > > Alternately, could we add a layering concept to the acceptance test >> harness? >> > There could be "precious" tests which run every time and there could be >> > other tests which are part of a collection from which a few tests are >> > selected randomly for execution every time. >> >> Yes this is possible. >> >> > is there a way to make the acceptance test harness run >> > in a massively parallel fashion >> >> Yes. Does not help with the flakiness, and still wastes a tremendous >> amount of cloud machine time. >> >> > Would we consider a policy that flaky tests are deleted, rather than > tolerated? Possibly with an email notification to each committer that > modified any line in the deleted test? > > I've understood that Kent Beck has mentioned that they delete flaky tests > at facebook, rather than just tolerating them. Seems dramatic, but > increases trust in the acceptance test results, and encourages moving tests > from the ATH into JenkinsRule or other locations where they may be easier > to diagnose. > > Mark Waite > > >> > As a safety check of that concept, did any of the current acceptance >> tests >> > detect the regression when run with Jenkins 2.80 (or Jenkins 2.80 RC)? >> >> Yes. >> >> > Is there a JenkinsRule test which could reasonably be written to test >> for >> > the conditions that caused the bug in Jenkins 2.80? >> >> Not really; that particular issue was unusual, since it touched on the >> setup wizard UI which is normally suppressed by test harnesses. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0LJ6z%3DTiQe8Rt9P_D7RpZDEHt3XAWQxoq%3Dd0dVAD%2BG%3Dw%40mail.gmail.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx%2Bu%2BzdH0Zg%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx%2Bu%2BzdH0Zg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Sent from my phone -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxRdYEOxD4gn7CxDS5nRK9PPD_tCzwGzsTS_2gg%2BRvqdg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
