I'd be a big +1 on this as well. It makes sense to have the harness as a separate project consumed by Jenkins core and plugins for their testing. A nice side effect is that we'd be able to actually move the core tests currently in jenkins.git/test/src/test/... into jenkins.git/core/src/test/..., I think, which would just make things feel much more organic.
A. On Fri, Jan 8, 2016 at 11:01 AM, Mark Waite <mark.earl.wa...@gmail.com> wrote: > I like the idea of moving JenkinsRule outside jenkins core very much. The > git plugin and git client plugin intentionally support older Jenkins > versions, but that means their test infrastructure is also limited to those > older Jenkins versions. I'd much rather be able to use the latest Jenkins > test infrastructure to create better tests. > > That would also provide me more incentive to attempt fixes to test > infrastructure. Today, if the test classes have a problem, there is very > little chance that I'll investigate the problem in the test classes, > because I can't access updates to those test classes anyway. > > Mark Waite > > On Fri, Jan 8, 2016 at 10:16 AM Jesse Glick <jgl...@cloudbees.com> wrote: > >> Today the Jenkins functional test harness, containing `JenkinsRule` >> and its ilk¹, is built from the Jenkins core repository, in >> `test/src/main/`. That is fine for functional tests of core itself >> (`test/src/test/`). But this scheme is undesirable for functional >> tests in plugins, for two reasons. >> >> One, whenever a fix is made to the test harness itself, such as my >> recent PR 1967², it does not become available to plugins until they >> switch their core baseline to the version including the fix. That >> could be months away, even if the fix is backported to an LTS branch >> as is occasionally done. Sometimes it is possible to work around the >> lack of the fix in plugin tests³, but even when possible this is >> awkward. >> >> Second, incompatible changes to the test harness are sometimes >> necessary or desirable. We try hard to avoid incompatible changes to >> core itself—these could break user installations—but incompatibilities >> of tests is a price paid only by plugin developers, usually >> manageable. In particular, PR 1774⁴ changed some details of HtmlUnit >> usage. Now if you upgrade your plugin’s core baseline to a version >> after that point (generally 1.625.x), you update your tests to >> match—fine. The problem comes as we try to expand usage of >> `plugin-compat-tester`⁵ to hunt for core regressions. To verify that a >> plugin’s tests still pass against a newer core, this tool updates the >> baseline in a working copy of the POM. But if the tests are still >> written against the old forked HtmlUnit, they may no longer compile; >> and a single version of a test cannot always be made to work against >> either version of HtmlUnit. You can see an example of this dilemma in >> the `copyartifact` plugin⁶. >> >> My proposal is to start treating the harness as an independently >> versioned test library, in its own GitHub repository. >> `jenkinsci/jenkins/test/src/main/` would be extracted there (with >> history intact). `test/src/test/` core functional tests would pick it >> up as a binary dependency, as plugin tests already do. The advantage >> would be that we could iterate quickly on new features or fixes to the >> harness, cutting a Maven release at any time. Core and each plugin >> would decide which version to pick up; whenever you wish to update to >> the latest version, you can edit your tests in the same commit to >> reflect any required changes, or to take advantage of new features. >> `plugin-compat-tester` would leave this version untouched. >> >> In principle there could be a change to core which requires a >> synchronized matching change to the test harness, which would become >> more difficult to handle in such a system, but I am not aware of any >> examples.⁷ >> >> After a split it should be possible to do further refactorings that >> would make for smaller artifacts and better modularity. Currently most >> of the size of the test harness artifact is consumed by bundling four >> (!) copies of Maven, plus one copy of Ant, which are only relevant to >> plugins actually testing builds using these tools. The harness also >> has fixed dependencies on a number of plugins which were long since >> split out of core—`subversion`, `maven-plugin`, `matrix-project`, >> etc.—which thickens the `test`-scope dependency tree for every plugin >> and can cause weird version resolution problems that complicate test >> development and maintenance. >> >> I am sending this proposal verbally rather than in a PR since >> cross-repository refactorings like this are not easily handled with >> PRs, especially since a new artifact release is required to get >> everything working. There are some details that will only fall out of >> an actual attempt to do it, but I wanted to get general agreement that >> this is a good direction before starting on things like the slow Git >> command to extract a subdirectory with history. >> >> For context, there is also some work being prepared to create a new >> plugin POM which, unlike the current one coversioned with core⁸, would >> be in its own repository and independently versioned, with the core >> version simply being a Maven property. Stephen Connolly and I have >> long discussed the possibility—CloudBees proprietary plugins do >> exactly this, and it is much more comfortable to work with as a plugin >> developer and from `plugin-compat-tester`—but there is finally some >> movement. The idea would be that with a refactored test harness, your >> plugin POM would specify both the versions of core and the test >> harness independently. >> >> WDYT? >> >> >> ¹ >> https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234 >> ² https://github.com/jenkinsci/jenkins/pull/1967 >> ³ https://github.com/jenkinsci/workflow-plugin/pull/277 >> ⁴ https://github.com/jenkinsci/jenkins/pull/1774 >> ⁵ https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester >> ⁶ https://github.com/jenkinsci/copyartifact-plugin/pull/76 >> ⁷ >> https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main >> includes >> https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd >> that would imply a newer core dep; this part of the patch was very low >> priority. >> https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55 >> and >> https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd >> handle core incompatibilities but are harmless to use with older >> cores. >> ⁸ >> https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-dev+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOacnvj1wJBEKCkzt_bzr5U8arkjo7Kgvi_2-XuTbWdhLA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
