> Am 28.09.2017 um 20:51 schrieb Stephen Connolly > <[email protected]>: > > > On Thu 28 Sep 2017 at 19:11, Mark Waite <[email protected] > <mailto:[email protected]>> wrote: > On Thu, Sep 28, 2017 at 11:43 AM Jesse Glick <[email protected] > <mailto:[email protected]>> wrote: > On Thu, Sep 28, 2017 at 10:42 AM, Mark Waite <[email protected] > <mailto:[email protected]>> wrote: > > Do we have any way of associating historical acceptance test harness > > failures to a cause of those failures? > > Not that I am aware of. > > > could such data be gathered from some sample of recent runs of the > > acceptance test harness, so that we know which of the tests have been most > > helpful in the recent past? > > Maybe. I doubt things are in good enough condition to do that kind of > analysis. AFAIK none of the CI jobs running the ATH have ever had a > single stable run, so there is not really a baseline. > > > Ah, so that means the current acceptance test harness is unlikely to be > trusted in its entirety by anyone. A failure in the entire acceptance test > harness is probably only used rarely to stop a release. > > I support deleting tests of questionable value. > > I don’t want to knock the contributors to the ATH, but my long standing view > is that writing good acceptance tests is a high skill and not something that > can be easily crowdsourced. >
But writing code without tests can crowdsourced? > Acceptance tests are, by their nature, among the slowest tests. > > My long standing view of the ATH is that there is a fundamental flaw in using > the same harness to drive as to verify *because* any change to that harness > has the potential to invalidate any tests using the modified harness... it is > all too easy to turn a good test into a test giving a false positive... I > think an ATH that had two sets of tests, one driving by the filesystem / CLI > and verifying via the UI while the other drives by the UI and verified by the > filesystem / CLI would be much much stronger (and faster too). In most cases > you wouldn’t be changing the two harnesses at the same time, so a failing > test will indicate a failing test. > > We need an ATH that can be realistically run by humans in an hour or two... > trim the test cases to get there, moving the tested functionality higher up > to be tests that run faster, eg JenkinsRule or better yet pure unit tests. Who is we? The core team? Yes, you should create an ATH that detects core problems (subset of the current ATH). Create a test suite with 100 ATH tests and you will get your proposed runtime. But this should not affect the ATH for plugins. You will never get a quality UI test suite for 1500 plugins in an hour. If a plug-in author crates a suite with 10-50 tests so what. This is up to the plugin author. (We discussed it during the initial days of the ATH if it wouldn’t be better to have the framework and page objects in the ATH only and the individual tests in the plugins. Maybe we should reconsider this topic again…) > > That’s just my view, feel free to ignore. (Because some people confuse my > interaction style I am going to try and clarify) I’m bowing out unless > someone specifically asks me for clarification on my recommendations or I see > people miss-representing my opinion... this is normally the way I interact... > just people have a real habit of dragging me back in and that makes me look > like someone who rabbits on and on. > > > > Alternately, could we add a layering concept to the acceptance test harness? > > There could be "precious" tests which run every time and there could be > > other tests which are part of a collection from which a few tests are > > selected randomly for execution every time. > > Yes this is possible. > > > is there a way to make the acceptance test harness run > > in a massively parallel fashion > > Yes. Does not help with the flakiness, and still wastes a tremendous > amount of cloud machine time. > > > Would we consider a policy that flaky tests are deleted, rather than > tolerated? Possibly with an email notification to each committer that > modified any line in the deleted test? > > I've understood that Kent Beck has mentioned that they delete flaky tests at > facebook, rather than just tolerating them. Seems dramatic, but increases > trust in the acceptance test results, and encourages moving tests from the > ATH into JenkinsRule or other locations where they may be easier to diagnose. > > Mark Waite > > > As a safety check of that concept, did any of the current acceptance tests > > detect the regression when run with Jenkins 2.80 (or Jenkins 2.80 RC)? > > Yes. > > > Is there a JenkinsRule test which could reasonably be written to test for > > the conditions that caused the bug in Jenkins 2.80? > > Not really; that particular issue was unusual, since it touched on the > setup wizard UI which is normally suppressed by test harnesses. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:jenkinsci-dev%[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0LJ6z%3DTiQe8Rt9P_D7RpZDEHt3XAWQxoq%3Dd0dVAD%2BG%3Dw%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0LJ6z%3DTiQe8Rt9P_D7RpZDEHt3XAWQxoq%3Dd0dVAD%2BG%3Dw%40mail.gmail.com>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx%2Bu%2BzdH0Zg%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx%2Bu%2BzdH0Zg%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxRdYEOxD4gn7CxDS5nRK9PPD_tCzwGzsTS_2gg%2BRvqdg%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxRdYEOxD4gn7CxDS5nRK9PPD_tCzwGzsTS_2gg%2BRvqdg%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/716A4677-29EF-45CB-A895-2F9EC1BD66C4%40gmail.com. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: Message signed with OpenPGP
