After the fiasco of Jenkins 2.80 some of us have been discussing ways to speed up `acceptance-test-harness` to the point where it would be practical to run all or part of it against core more frequently, perhaps even in core PR builds.
Part of the problem is that the test infrastructure has a ton of overhead. One suggestion was to use PhantomJS rather than a live browser like Firefox during typical test runs. I suggested optimizing the plugin upload mechanism in `LocalController`, perhaps using a mock local update center URL to avoid network downloads (as well as simplifying the way `LOCAL_JARS` and similar options work). Part of the problem is also that there are just a lot of tests, some of questionable value. https://github.com/jenkinsci/acceptance-test-harness/blob/0e3d03474c28c047b365d1f29b621af32eb7918c/src/test/java/plugins/JobDslPluginTest.java#L1537-L1563 is an example of why a full ATH run is so ridiculously slow. On one CI builder this single test case (there are 61 of them in the test suite for this one plugin) took 2m45s. To test what? That you can run a DSL script and it produces a `View` with certain matching jobs? This could run via `JenkinsRule` in a few seconds and be equally effective in catching regressions. (Probably more so, since it would be part of the plugin repo and thus run on every plugin PR, catching mistakes in plugin code. There is a smaller chance that a fundamental change in core would break this plugin functionality; that is what `plugin-compat-tester` is for.) We need to just delete stuff like this and make sure ATH is just testing the UI and critical (“smoke test”) functionality. Think of test harnesses in a pyramid: You can have as many unit tests as you want, which test individual methods or small self-contained components. They should run more or less instantaneously, and behave essentially the same on any platform. You should have a bunch of `JenkinsRule` “functional” tests, which are slower to run (a few seconds) and a little flakier, but which start up a real Jenkins server and test your plugin’s functionality in a more realistic way including extensions and background threads and project builds and settings and everything. These can even start (open-source Linux) external services using `docker-fixtures` where necessary, check basic aspects of HTML page rendering including form submissions with HtmlUnit, run CLI commands, etc. Finally you should have a handful of carefully chosen “acceptance” tests which are harder to develop, very slow to run, and often flaky, but provide a convincing demonstration that an entire user-level feature truly works in a realistic environment including production-style plugin and class loading (perhaps even contacting outside services) and driven entirely from the same kind of gestures a user would make in a web browser. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0UfKN8M1JF6pp7qdoKjo5SSHVwTcv_XVvbsGycK_tjPw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
