Lucie, thank you for your excellent summary of the current situation! Since only Mark responded so far this is an indication that actually only a few Jenkins developers are using ATH right now. I also think, that using ATH in the current way makes not much sense anymore.
As you suggested, a valid option would be to split the two groups of tests (smoke tests and plugin specific tests) also physically. This would make much sense for me, too. Having the ATH tests for my plugins in my own plugin would improve my development cycle: The tests could be automatically started in our CI server as soon as I push some changes (or a PR has been submitted). Then I also would get feedback if I broke something in the UI (at least if I run these tests in travis). On the other hand, this would require that we must be more carefully when we refactor ATH core classes since now we won’t see compile errors (due to API changes) immediately. And if plugin authors do not really care about these test in their plugins then they can feel free to delete them. However, this approach makes LTS testing more difficult. At least in theory - as far as I understood your post, in the current state ATH is not helpful in ATH testing anymore. > Am 29.11.2018 um 00:15 schrieb Mark Waite <[email protected]>: > > My replies are inline. I'm a plugin maintainer. I am deeply attached to > automated tests, especially for the plugins I maintain. > > I have not yet had a positive experience with automated tests that use a web > browser to perform those tests. I've been in software development for a long > time. I've seen several attempts, but no personal positive experiences yet. > > My replies are based on those biases. > > On Wed, Nov 28, 2018 at 6:48 AM Lucie Votypkova <[email protected] > <mailto:[email protected]>> wrote: > Hello, > I would like to start discussion about ATH selenium testing. After year of > attempts to help with maintenance and LTS testing, we are dealing with > problem that maintenance is becoming very time demanding and added value of > it is not sufficient. The tests were created for replacing manual testing LTS > with the most used plugins, showing that changes in Jenkins core does not > break widely used plugins and of course testing of Jenkins UI. In current > state is better then manual testing, but it is still time demanding to keep > the step with UI changes in plugins and core and in the same time deal with > browsers, driver, selenium issues and bugs in Jenkins (and its plugins). Due > to a lot of constant failures (which are not connected with UI issues or > integration of Jenkins with plugins in many cases), it can not be used for > watching impact of every Jenkins release on current plugins and it does not > save a lot of time in LTS testing (if we consider the time for checking > constant failures plus time for maintenance) . > > I collected the main problems I had to deal with. And a few questions which > we should think about and suggestions what could help. Please feel free to > share your thoughts. > > > Problems > Maintenance is very time consuming. > It is revealed only for very small amount of bugs in core or plugins (most > failures are connected with changes in UI). > The most of revealed plugin bugs in last year can be detected by functional > testing. > Some failures which are real bugs in plugins have a very low priority because > it has a small impact on real user. But it sometime bloks whole testing and > building workaround would not fit the main purpose - use jenkins as a real > user do. This causes that there are real failures (sometime in the beginning > of the tests) which will not be fixed in near future. > There are a lot of tests failures because of changes in plugin UI. When it > happens with widely used plugin which is a dependency of many other popular > plugins (credentials, for example), the ATH tests are useless until this > changes are fixed. For example: > https://issues.jenkins-ci.org/browse/JENKINS-49026 > <https://issues.jenkins-ci.org/browse/JENKINS-49026> > Vast majority of breakage caused by UI changes is mostly fixed by maintainers > of ATH because plugins/core contributors are not generally aware of ATH > existence, that they have broke some of the tests or even how to fix that. > Reliability of ATH suite is now stuck in a cycle: ATH is not reliable because > there are always quite a lot of failures and it never gives zero failures > result. The most failures are caused by changes in plugin UI or low priority > bugs in plugins which mainteners are not keen to fix. After a year attempts > to fix ATH failures and achieve +- zero failures at least for LTS and the > last versions of plugins, I realized that the issues appears faster than I am > able to fix them. Now I am able to keep with them but not to decrease their > amount. > Due to many failures caused by UI changes, plugin bugs, browser upgrade, > marionette/geckodriver bugs are not very useful tool for showing successful > integration new core versions (and LTS) with the most widely used plugins. > Currently to get such a measurement, it is necessary to verify about 30-50 > failures in ATH (if they are still caused only by plugin bugs, UI changes….). > Which is quite time consuming in comparing with situation when there is 5-10 > failures. > Important integration aspects has no coverage in automated testing? > The testsuite takes ~20 hours to complete. > > > Questions: > How to deal with tests that are broken (failing without detecting bug) for > long time and there is noone to fix the test? > How to deal with known bugs which breaks ATH test and do not seem to get > fixed in near future? > How to make maintainers of plugins to feel that it is beneficial to > participate and help with maintenance of ATH, ideally having them actively > engaged in making ATH compatible with their plugin changes? > > > Suggestions: > Avoid ATH tests that duplicates JTH tests in core/plugin? > Drawback: Would require PCT to be part of LTS certification. > Use only subset of ATH for LTS certification? > Drawback: even less attention to the second class citizens. > Guideline of removing fragile tests. > How to detect fragility? > Guideline for including (or excluding) plugin tests into ATH based on the > size of the userbase? > Guideline for reporting broken plugin tests to plugin/test maintainer > including test removal if not fixed in time? > What would be the deadline? > Avoid trivial tests in favor of meaningful use-cases > Will cause collapsing/removal of many tests. > Guideline for maximal number of tests / maximal runtime per plugin? > Suggest perform ATH testing for pull requests of plugins which are stable for > long time in ATH by changing Jenkinsfile (only tests for given plugin will be > performed, not whole ATH) > It will definitely make ATH more visible make people more cooperate, but! > Will it be acceptable from time perspective (ATH is running for long time)? > > No, it won't be acceptable from a time perspective (at least it seems > unlikely to me). PCT tests were added to the git plugin. The cycle time for > pull request evaluation increased beyond acceptable limits (from 15 minutes > to an hour or more if I remember correctly). That might have been acceptable > except for the other issues (described later in this message) > > Will it complicate the pull request? ATH tests does not fail only because of > bugs in given plugin, it does not have to be connected with given pull > request. > > PCT tests were added to the git plugin. Random failures happened when there > were no relevant code changes. As a plugin maintainer, I had no interest in > investigating the failure of a long-running test which fails for unrelated or > transient reasons. I didn't see the benefit of investigating those failures. > They were not telling me something especially useful about the problem they > were reporting. Time researching an unrelated or transient failure is time > invested in something that rarely helps the plugin I maintain and rarely > helps the Jenkins community. > > I've shifted my "end to end" testing for the git plugin to use one > multibranch pipeline job for each bug that I'm investigating (see > jenkins-bugs <https://github.com/MarkEWaite/jenkins-bugs>). I use the > Jenkins Pipeline code and the build tools of my choice to perform checks that > specific conditions of the specific bug are satisfied or not satisfied. > However, those end to end tests involve no web browser and tend to remain > tightly focused on verifying a very specific bug, not an entire use case. > > Use case verification still requires that I test interactively. > > Can those tests be moved to plugin repositories altogether? > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > -- > Thanks! > Mark Waite > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGmcygJqsvwmVgxEuTYtWfy8zuQ7LoAzAj%2BKy0AN%3DAApw%40mail.gmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGmcygJqsvwmVgxEuTYtWfy8zuQ7LoAzAj%2BKy0AN%3DAApw%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3E74D764-768B-4825-B6E2-B42B6E654D7B%40gmail.com. For more options, visit https://groups.google.com/d/optout.
