I'm think that a Jenkins job which asserts the conditions related to a bug report is a regression test. I admit that the Jenkins jobs I create are not Selenium tests, and they do not require a web browser in order to run. They are definitely not able to exercise JavaScript, and they do not evaluate behavior across multiple browsers. If the goal is to check JavaScript or to check browser compatibility, then Jenkins jobs are a poor choice.
If the goal is to duplicate a bug report, and reuse the steps that duplicated the bug report for later regression testing, then I think a Jenkins job is simpler to create, easier to maintain, and easier to diagnose than a Selenium test. Ulrich, if you are interested in some "test drive" bug reports from the git plugin or the git client plugin, you can review how I use the technique to report and duplicate bugs in: - JENKINS-42204 <https://issues.jenkins-ci.org/browse/JENKINS-42204> - NPE due to recent GitSCMSource change - JENKINS-41948 <https://issues.jenkins-ci.org/browse/JENKINS-41948> - NoSuchMethodError scanning multi-branch project - JENKINS-41867 <https://issues.jenkins-ci.org/browse/JENKINS-41867> - IllegalStateException scanning multi-branch project - JENKINS-41785 <https://issues.jenkins-ci.org/browse/JENKINS-41785> - Pipeline status icon no longer shows worst status If your students would like to try to duplicate bugs that I think are interesting, I would suggest starting with: - JENKINS-38860 <https://issues.jenkins-ci.org/browse/JENKINS-38860> - renamed submodules fail checkout - JENKINS-41553 <https://issues.jenkins-ci.org/browse/JENKINS-41553> - submodule defs read from wrong location - JENKINS-42004 <https://issues.jenkins-ci.org/browse/JENKINS-42004> - jobs triggered unnecessarily (scenario 1) - JENKINS-41906 <https://issues.jenkins-ci.org/browse/JENKINS-41906> - jobs triggered unnecessarily (scenario 2) - JENKINS-40733 <https://issues.jenkins-ci.org/browse/JENKINS-40733> - notifyCommit provides wrong parameters - JENKINS-40352 <https://issues.jenkins-ci.org/browse/JENKINS-40352> - duplicate changesets in pipeline jobs (and other jobs) - JENKINS-39621 <https://issues.jenkins-ci.org/browse/JENKINS-39621> - infinite loop of job rebuilds Those bug reports are likely well suited to creating one or more jobs to show that the bug exists. If they want to use the infrastructure already provided in the lts-with-plugins branch <https://github.com/MarkEWaite/docker/tree/lts-with-plugins> of my docker repository <https://github.com/MarkEWaite/docker>, they're welcome to do that. If you decide that the technique works well enough for the students, then you can consider attempting other bugs from the git-plugin and git-client-plugin projects, most recent bugs first. I suspect other SCM system bugs (Subversion, etc.) are reasonably well suited to the same technique. I've also found that it is quite powerful to create a pipeline based branch per bug verification job so that Jenkins will automatically create the job and execute it. You can see examples of that technique in my jenkins-bugs repository <https://github.com/MarkEWaite/jenkins-bugs> (which is referenced in several different ways from the docker instance). That has the benefit that I don't have to commit anything to the docker repository, and the Jenkins pipeline code still creates and runs a job per bug. I'm also OK if you decide it is better to submit pull requests to the acceptance test harness. The acceptance test harness has the compelling benefit that there are many more people creating Selenium tests with the acceptance test harness, and there are many more locations where those tests can be executed. In my case, I don't need the flexibility of a Selenium test, and I want the easier diagnose and trivial parallel execution that comes from running a Jenkins instance with multiple slaves and with tests defined as Jenkins jobs. Thanks! Mark Waite P.S. I have pipeline utility functions assertLogContains and assertLogDoesNotContain to search for specific strings in the log files. When one of those functions finds something unexpected, it marks the job as "Unstable". I can then search for all failed jobs and all unstable jobs in my Jenkins instance to locate test failures. Because it is a docker instance, I can spread the execution load trivially across multiple agents by adding or deleting agents. I get interesting (and useful) test randomization by choosing a mix of operating systems (Windows, CentOS 6. CentOS 7, Debian 7, Debian 8, Ubuntu 14, Ubuntu 16, etc.) and git versions. On Fri, Feb 17, 2017 at 5:47 AM Jesse Glick <jgl...@cloudbees.com> wrote: > I would also much rather see a genuine regression test. If some aspect of > ATH is too weak to reproduce the bug properly, then that is what needs > fixing. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2eGwB-hbFXtJa%2BJE72HbTqE6t8iTqYQO_iRL-SBarbBw%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2eGwB-hbFXtJa%2BJE72HbTqE6t8iTqYQO_iRL-SBarbBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGNukWHg8s2qx-GiCEN%2BPRKLOdRQ1f3UM50J4%2BgBB9r8A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
