The original "selling points" for me with the ATH was that it provided ways to test plugin functionality together with other plugins already installed. Running plugin integration tests with "just" the test dependencies is one thing but will it still work when 250 other plugins are installed alongside this one. In that scenario even the simplest "happy path" plugin test is worth it in my mind. The same i believe is valid from a validating core perspective. It's one thing to test a core functionality works with one plugin installed, but does it still work with 250 other plugins? Or at least the recommended plugin set installed? No matter where the fault is when something breaks, the overall "product experience" will be seen as broken if something doesn't work with core + the recommended plugins imho.
Now getting that mode to work in the ATH is not simple so I guess very few (if anyone) are running it in that way, I don't even know if that run mode even works any more. So if we run a curated set of tests with the recommended plugins pre-installed we'll at least lower/remove the plugin install part of the test time, but then we won't be testing the setup wizard, and the reason for why this discussion is happening would be void. But a separate run of "just" the install experience oriented tests could be run on the side with a blank jenkinsHome :) /B 2017-09-29 0:56 GMT+02:00 Ullrich Hafner <[email protected]>: > > Am 28.09.2017 um 20:51 schrieb Stephen Connolly < > [email protected]>: > > > On Thu 28 Sep 2017 at 19:11, Mark Waite <[email protected]> wrote: > >> On Thu, Sep 28, 2017 at 11:43 AM Jesse Glick <[email protected]> >> wrote: >> >>> On Thu, Sep 28, 2017 at 10:42 AM, Mark Waite <[email protected]> >>> wrote: >>> > Do we have any way of associating historical acceptance test harness >>> > failures to a cause of those failures? >>> >>> Not that I am aware of. >>> >>> > could such data be gathered from some sample of recent runs of the >>> > acceptance test harness, so that we know which of the tests have been >>> most >>> > helpful in the recent past? >>> >>> Maybe. I doubt things are in good enough condition to do that kind of >>> analysis. AFAIK none of the CI jobs running the ATH have ever had a >>> single stable run, so there is not really a baseline. >>> >>> >> Ah, so that means the current acceptance test harness is unlikely to be >> trusted in its entirety by anyone. A failure in the entire acceptance test >> harness is probably only used rarely to stop a release. >> >> I support deleting tests of questionable value. >> > > I don’t want to knock the contributors to the ATH, but my long standing > view is that writing good acceptance tests is a high skill and not > something that can be easily crowdsourced. > > > But writing code without tests can crowdsourced? > > Acceptance tests are, by their nature, among the slowest tests. > > My long standing view of the ATH is that there is a fundamental flaw in > using the same harness to drive as to verify *because* any change to that > harness has the potential to invalidate any tests using the modified > harness... it is all too easy to turn a good test into a test giving a > false positive... I think an ATH that had two sets of tests, one driving by > the filesystem / CLI and verifying via the UI while the other drives by the > UI and verified by the filesystem / CLI would be much much stronger (and > faster too). In most cases you wouldn’t be changing the two harnesses at > the same time, so a failing test will indicate a failing test. > > We need an ATH that can be realistically run by humans in an hour or > two... trim the test cases to get there, moving the tested functionality > higher up to be tests that run faster, eg JenkinsRule or better yet pure > unit tests. > > > Who is we? The core team? Yes, you should create an ATH that detects core > problems (subset of the current ATH). Create a test suite with 100 ATH > tests and you will get your proposed runtime. > But this should not affect the ATH for plugins. You will never get a > quality UI test suite for 1500 plugins in an hour. If a plug-in author > crates a suite with 10-50 tests so what. This is up to the plugin author. > > (We discussed it during the initial days of the ATH if it wouldn’t be > better to have the framework and page objects in the ATH only and the > individual tests in the plugins. Maybe we should reconsider this topic > again…) > > > That’s just my view, feel free to ignore. (Because some people confuse my > interaction style I am going to try and clarify) I’m bowing out unless > someone specifically asks me for clarification on my recommendations or I > see people miss-representing my opinion... this is normally the way I > interact... just people have a real habit of dragging me back in and that > makes me look like someone who rabbits on and on. > > >> > Alternately, could we add a layering concept to the acceptance test >>> harness? >>> > There could be "precious" tests which run every time and there could be >>> > other tests which are part of a collection from which a few tests are >>> > selected randomly for execution every time. >>> >>> Yes this is possible. >>> >>> > is there a way to make the acceptance test harness run >>> > in a massively parallel fashion >>> >>> Yes. Does not help with the flakiness, and still wastes a tremendous >>> amount of cloud machine time. >>> >>> >> Would we consider a policy that flaky tests are deleted, rather than >> tolerated? Possibly with an email notification to each committer that >> modified any line in the deleted test? >> >> I've understood that Kent Beck has mentioned that they delete flaky tests >> at facebook, rather than just tolerating them. Seems dramatic, but >> increases trust in the acceptance test results, and encourages moving tests >> from the ATH into JenkinsRule or other locations where they may be easier >> to diagnose. >> >> Mark Waite >> >> >>> > As a safety check of that concept, did any of the current acceptance >>> tests >>> > detect the regression when run with Jenkins 2.80 (or Jenkins 2.80 RC)? >>> >>> Yes. >>> >>> > Is there a JenkinsRule test which could reasonably be written to test >>> for >>> > the conditions that caused the bug in Jenkins 2.80? >>> >>> Not really; that particular issue was unusual, since it touched on the >>> setup wizard UI which is normally suppressed by test harnesses. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google. >>> com/d/msgid/jenkinsci-dev/CANfRfr0LJ6z%3DTiQe8Rt9P_ >>> D7RpZDEHt3XAWQxoq%3Dd0dVAD%2BG%3Dw%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google. >> com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx% >> 2Bu%2BzdH0Zg%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtG5DGH3vHt_bwN972EeHeEcPSJekd0dZ%2BQKx%2Bu%2BzdH0Zg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google. > com/d/msgid/jenkinsci-dev/CA%2BnPnMxRdYEOxD4gn7CxDS5nRK9PPD > _tCzwGzsTS_2gg%2BRvqdg%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxRdYEOxD4gn7CxDS5nRK9PPD_tCzwGzsTS_2gg%2BRvqdg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jenkinsci-dev/716A4677-29EF-45CB-A895-2F9EC1BD66C4%40gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/716A4677-29EF-45CB-A895-2F9EC1BD66C4%40gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Robert Sandell *Software Engineer* *CloudBees Inc.* -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALzHZS1jv6gCkRLpQgfu_RL0K968sjyqS5X6UnyAx3v7u6-bcA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
