My replies are inline. I'm a plugin maintainer. I am deeply attached to automated tests, especially for the plugins I maintain.
I have not yet had a positive experience with automated tests that use a web browser to perform those tests. I've been in software development for a long time. I've seen several attempts, but no personal positive experiences yet. My replies are based on those biases. On Wed, Nov 28, 2018 at 6:48 AM Lucie Votypkova <[email protected]> wrote: > Hello, > > I would like to start discussion about ATH selenium testing. After year of > attempts to help with maintenance and LTS testing, we are dealing with > problem that maintenance is becoming very time demanding and added value of > it is not sufficient. The tests were created for replacing manual testing > LTS with the most used plugins, showing that changes in Jenkins core does > not break widely used plugins and of course testing of Jenkins UI. In > current state is better then manual testing, but it is still time demanding > to keep the step with UI changes in plugins and core and in the same time > deal with browsers, driver, selenium issues and bugs in Jenkins (and its > plugins). Due to a lot of constant failures (which are not connected with > UI issues or integration of Jenkins with plugins in many cases), it can not > be used for watching impact of every Jenkins release on current plugins and > it does not save a lot of time in LTS testing (if we consider the time for > checking constant failures plus time for maintenance) . I collected the > main problems I had to deal with. And a few questions which we should think > about and suggestions what could help. Please feel free to share your > thoughts. > > > Problems > > - > > Maintenance is very time consuming. > - > > It is revealed only for very small amount of bugs in core or plugins > (most failures are connected with changes in UI). > - > > The most of revealed plugin bugs in last year can be detected by > functional testing. > - > > Some failures which are real bugs in plugins have a very low priority > because it has a small impact on real user. But it sometime bloks whole > testing and building workaround would not fit the main purpose - use > jenkins as a real user do. This causes that there are real failures > (sometime in the beginning of the tests) which will not be fixed in near > future. > - > > There are a lot of tests failures because of changes in plugin UI. > When it happens with widely used plugin which is a dependency of many other > popular plugins (credentials, for example), the ATH tests are useless until > this changes are fixed. For example: > https://issues.jenkins-ci.org/browse/JENKINS-49026 > - > > Vast majority of breakage caused by UI changes is mostly fixed by > maintainers of ATH because plugins/core contributors are not generally > aware of ATH existence, that they have broke some of the tests or even how > to fix that. > - > > Reliability of ATH suite is now stuck in a cycle: ATH is not reliable > because there are always quite a lot of failures and it never gives zero > failures result. The most failures are caused by changes in plugin UI or > low priority bugs in plugins which mainteners are not keen to fix. After a > year attempts to fix ATH failures and achieve +- zero failures at least for > LTS and the last versions of plugins, I realized that the issues appears > faster than I am able to fix them. Now I am able to keep with them but not > to decrease their amount. > - > > Due to many failures caused by UI changes, plugin bugs, browser > upgrade, marionette/geckodriver bugs are not very useful tool for showing > successful integration new core versions (and LTS) with the most widely > used plugins. Currently to get such a measurement, it is necessary to > verify about 30-50 failures in ATH (if they are still caused only by plugin > bugs, UI changes….). Which is quite time consuming in comparing with > situation when there is 5-10 failures. > - > > Important integration aspects has no coverage in automated testing? > - > > The testsuite takes ~20 hours to complete. > > > > Questions: > > - > > How to deal with tests that are broken (failing without detecting bug) > for long time and there is noone to fix the test? > - > > How to deal with known bugs which breaks ATH test and do not seem to > get fixed in near future? > - > > How to make maintainers of plugins to feel that it is beneficial to > participate and help with maintenance of ATH, ideally having them actively > engaged in making ATH compatible with their plugin changes? > > > > Suggestions: > > - > > Avoid ATH tests that duplicates JTH tests in core/plugin? > - > > Drawback: Would require PCT to be part of LTS certification. > - > > Use only subset of ATH for LTS certification? > - > > Drawback: even less attention to the second class citizens. > - > > Guideline of removing fragile tests. > - > > How to detect fragility? > - > > Guideline for including (or excluding) plugin tests into ATH based on > the size of the userbase? > - > > Guideline for reporting broken plugin tests to plugin/test maintainer > including test removal if not fixed in time? > - > > What would be the deadline? > - > > Avoid trivial tests in favor of meaningful use-cases > - > > Will cause collapsing/removal of many tests. > - > > Guideline for maximal number of tests / maximal runtime per plugin? > > > - > > Suggest perform ATH testing for pull requests of plugins which are > stable for long time in ATH by changing Jenkinsfile (only tests for given > plugin will be performed, not whole ATH) > - > > It will definitely make ATH more visible make people more > cooperate, but! > - > > Will it be acceptable from time perspective (ATH is running for > long time)? > > No, it won't be acceptable from a time perspective (at least it seems unlikely to me). PCT tests were added to the git plugin. The cycle time for pull request evaluation increased beyond acceptable limits (from 15 minutes to an hour or more if I remember correctly). That might have been acceptable except for the other issues (described later in this message) > > - > > Will it complicate the pull request? ATH tests does not fail > only because of bugs in given plugin, it does not have to be > connected with > given pull request. > > PCT tests were added to the git plugin. Random failures happened when there were no relevant code changes. As a plugin maintainer, I had no interest in investigating the failure of a long-running test which fails for unrelated or transient reasons. I didn't see the benefit of investigating those failures. They were not telling me something especially useful about the problem they were reporting. Time researching an unrelated or transient failure is time invested in something that rarely helps the plugin I maintain and rarely helps the Jenkins community. I've shifted my "end to end" testing for the git plugin to use one multibranch pipeline job for each bug that I'm investigating (see jenkins-bugs <https://github.com/MarkEWaite/jenkins-bugs>). I use the Jenkins Pipeline code and the build tools of my choice to perform checks that specific conditions of the specific bug are satisfied or not satisfied. However, those end to end tests involve no web browser and tend to remain tightly focused on verifying a very specific bug, not an entire use case. Use case verification still requires that I test interactively. > > - > > Can those tests be moved to plugin repositories altogether? > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGmcygJqsvwmVgxEuTYtWfy8zuQ7LoAzAj%2BKy0AN%3DAApw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
