On Thu, Sep 28, 2017 at 6:31 PM, Ullrich Hafner <ullrich.haf...@gmail.com> wrote: > In order to verify core PRs just use the smoke tests. You can select the > corresponding tests on your own by changing the annotated tests.
Yes I have seen this, though I think there is no one really keeping track of which tests are annotated in this way and making sure that the set is sane. The way ATH is structured it is not suitable for running in core PRs I think: tests could start failing with no relation to the core change under test. Even if you pinned a specific revision of `acceptance-test-harness`, you would still be puling some unknown version of each plugin from the update center. Even if you exclude all tests with `@WithPlugins` (making running the tests far less useful), many of them have intrinsic external dependencies that we can expect to be flaky. OTOH it makes sense to run a defined set of tests on a regular basis, having someone designated to watch over the results, and ensuring that weekly releases with known failures are not published. > we have one test for two of our plugins, or we spend 30 seconds for each > plugin. This is a ridiculously few number of tests! If that were the only test coverage, sure. But the vast majority of tests are functional tests in plugin repositories. ATH contributes relatively little to our automated test coverage. > I use these tests (not all, just the tests for my plugins) before I make a > plugin release. And in fact the tests for the Analysis suite are particularly slow, so this is a good example of my point. A handful of them test the UI but most seem that they could have been written more naturally as functional tests. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2qM%3Dk6Kh2N37BQNOHzPHE4L3CT1U2Rw1CsJ3S_ZcLC4A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
