> Am 28.09.2017 um 16:42 schrieb Mark Waite <[email protected]>: > > > [..] > > Do we have any way of associating historical acceptance test harness failures > to a cause of those failures? >
I found a lot of regressions in my plugins with theses tests, though I don’t have any numbers. > If not, could such data be gathered from some sample of recent runs of the > acceptance test harness, so that we know which of the tests have been most > helpful in the recent past? > I do not think that this is possible (or even required). > For example, I'd support deleting an acceptance test if it has never detected > a bug since it was created. This makes no sense. A good ATH test case is there to prevent a regression. So after what age would you like to remove a test that caused no failure? Two months, two years? > > Alternately, could we add a layering concept to the acceptance test harness? > There could be "precious" tests which run every time and there could be other > tests which are part of a collection from which a few tests are selected > randomly for execution every time. This is already possible. You just need someone of the core team that identifies which of the tests are essential for core features. Most ATH tests are testing plugin features. > > As another alternate, is there a way to make the acceptance test harness run > in a massively parallel fashion, with a test per machine, and hundreds of > machines running the tests all at once? > > You can have as many unit tests as you want, which test individual > methods or small self-contained components. They should run more or > less instantaneously, and behave essentially the same on any platform. > > You should have a bunch of `JenkinsRule` “functional” tests, which are > slower to run (a few seconds) and a little flakier, but which start up > a real Jenkins server and test your plugin’s functionality in a more > realistic way including extensions and background threads and project > builds and settings and everything. These can even start (open-source > Linux) external services using `docker-fixtures` where necessary, > check basic aspects of HTML page rendering including form submissions > with HtmlUnit, run CLI commands, etc. > > Finally you should have a handful of carefully chosen “acceptance” > tests which are harder to develop, very slow to run, and often flaky, > but provide a convincing demonstration that an entire user-level > feature truly works in a realistic environment including > production-style plugin and class loading (perhaps even contacting > outside services) and driven entirely from the same kind of gestures a > user would make in a web browser. > > > As a safety check of that concept, did any of the current acceptance tests > detect the regression when run with Jenkins 2.80 (or Jenkins 2.80 RC)? > > Is there a JenkinsRule test which could reasonably be written to test for the > conditions that caused the bug in Jenkins 2.80? > > Mark Waite > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:jenkinsci-dev%[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0UfKN8M1JF6pp7qdoKjo5SSHVwTcv_XVvbsGycK_tjPw%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0UfKN8M1JF6pp7qdoKjo5SSHVwTcv_XVvbsGycK_tjPw%40mail.gmail.com>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHGrvQtypR7UyxrmGqTt65NS2iq2p-_2oezHg6q%2B8QwKA%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHGrvQtypR7UyxrmGqTt65NS2iq2p-_2oezHg6q%2B8QwKA%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F70197EA-89BE-4029-B03D-38540ECD0D4C%40gmail.com. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: Message signed with OpenPGP
