[JIRA] (JENKINS-60295) jenkins-test-harness does not handle detached plugins properly

[robinverduijn.git...@gmail.com (JIRA)]

Title: Message Title

Robin Verduijn created an issue   Jenkins / JENKINS-60295 jenkins-test-harness does not handle detached plugins properly Issue Type: Bug Assignee: Oliver Gondža Attachments: jenkins-test-harness.zip, patch-2.176-to-2.190.diff, patch-skip-declared-plugins.diff, Screen Shot 2019-11-26 at 16.24.47.png, Screen Shot 2019-11-26 at 16.25.32.png Components: core, jenkins-test-harness Created: 2019-11-27 01:05 Priority: Minor Reporter: Robin Verduijn The jenkins-test-harness framework does not seem to handle Jenkins' detached plugins properly.  If one of the detached plugins bundled in the jenkins.war file is older than what is specified in a Gradle jenkinsPlugins dependency, both will get installed and when you try to run the local Jenkins test instance it complains that the older version is installed and refuses to install any dependencies that require the newer version.   This breaks tools like the Gradle JobDSL plugin as they are unable to specify a newer set of plugins to match a running Jenkins instance if some of those plugins are bundled at older versions in the jenkins-war artifact.   For instance, if you had a Jenkins instance running 2.190.2 with a set of plugins updated to be (more or less) current, and tried to use the jenkins-test-harness runner to execute DSL and create jobs on your Jenkins instance, it would complain that "SCM API Plugin version 2.4.1 is older than required. To fix, install version 2.6.3 or later." because 2.4.1 is what is bundled in the 2.190.2 jenkins-war file, even though a newer version should come in through a dependency on the latest Git plugin (e.g. Git 4.0.0).   When inspecting the test Jenkins instance spun up by the jenkins-test-harness, you will see that it has ended up installing BOTH versions of scm-api when it should have let the newer 2.6.3 version take precedence over 2.4.1. E.g. see this:       This can be reproduced with a simple Gradle project consisting of a build.gradle file, a sample job DSL, and a small class which uses the jenkins-test-harness to provision a test Jenkins instance. This can be found in the attached jenkins-test-harness.zip file, in the jenkins-test-harness/steps-2.190.2 folder. To reproduce it there, run the following Gradle command:   cd jenkins-teat-harness/steps-2.190.2 gradlew clean testJenkins   Also included in the attached archive is a similar project but for Jenkins 2.176.3. In that version, Git 4.0.0 is not yet available so the build.gradle file specifies 3.12.0 and the jenkins-war bundled detached-plugins match what's in the build.gradle so there is no mismatch, and the test runner works okay. The difference between the two can be found in patch-2.176-to-2.190.diff   It is possible to make the above testJenkins task run successfully by using a custom TestPluginManager which skips loading the declared-plugins from the jenkins-war artifact entirely. This does mean the caller is then responsible for providing enough plugins to make their test harness run correctly. An example of that working can be found in the attached .zip archive as well, in a folder jenkins-test-harness/steps-patched. The difference between that and the steps-2.190.2 folder can be found in patch-skip-declared-plugins.diff(notably, the addition of an additional plugin that is required and not loaded from jenkins-war anymore, and the specification of a custom plugin manager for the JenkinsRule which handles creating the test instance.)   I am not entirely sure what the proper fix here is. If the jenkins-test-harness allowed a flag to indicate whether to load the detached-plugins folder from inside the jenkins-war file, that would allow us to avoid loading them and instead rely on the specific versions we need to match our running Jenkins instance. We are running with a local patch which does this for us and it works.... okay. However, it also seems odd that the jenkins-test-harness would install an older plugin from jenkins-war's detached-plugins when a newer one is already specified in the jenkinsPlugins dependency section of a build.gradle file. If it were smarter and did not try to keep the older version installed, the problem would likely disappear altogether. A third possibility would be for jenkins-war not to bundle a too-old version of scm-api anymore. I suspect the version currently bundled in jenkins-war:2.190.2 might come in transitively as I could not find any evidence of its being explicitly included there, but I might have missed something. However, keeping jenkins-war always updated with newer plugins seems somewhat counterintuitive and it would seem more flexible for jenkins-test-harness to avoid loading older/outdated plugins if they were bundled in the detached-plugins folder.   While running the above "gradle testJenkins" custom task is an easy way to actually inspect the test instance provisioned through the test harness, it is of course also possible to run the test harness "the usual way". When run on the 2.176.3 version of the small sample project, everything is okay and it runs as follows:   $ gradlew clean dslUpdateJenkins --dryRun --jenkinsUrl=https://my-production-jenkins.io/ --jenkinsUser=xxx --jenkinsApiToken=xxxxx > Task :dslUpdateJenkins WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-all/2.4.15/423a17aeb2f64bc6f76e8e44265a548bec80fd42/groovy-all-2.4.15.jar) to method java.lang.Object.finalize() WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.25/8dacf9514f0c707cbbcdd6fd699e8940d42fb54e/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-jdk14/1.7.25/bccda40ebc8067491b32a88f49615a747d20082d/slf4j-jdk14-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] [main] INFO org.eclipse.jetty.util.log - Logging initialized @709ms to org.eclipse.jetty.util.log.Slf4jLog Test timeout disabled. === Starting Run Job DSL Task [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - jetty-9.4.5.v20170502 [Executing Run Job DSL Task] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Scavenging every 660000ms [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@93f432e{/jenkins,file:///Users/jenkins/Desktop/jenkins-test-harness/steps-2.176.3/build/jenkins-for-test/,AVAILABLE}{/Users/jenkins/Desktop/jenkins-test-harness/steps-2.176.3/build/jenkins-for-test} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@21f459fc{HTTP/1.1,[http/1.1]}{localhost:55162} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - Started @3117ms [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - jetty-9.4.5.v20170502 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Scavenging every 600000ms [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@4288d98e{/,null,AVAILABLE} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@7c601d50{HTTP/1.1,[http/1.1]}{0.0.0.0:55164} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - Started @9469ms Installed plugins: [jdk-tool:1.0, command-launcher:1.0, script-security:1.56, jaxb:2.3.0, workflow-durable-task-step:2.22, bouncycastle-api:2.16.0, matrix-project:1.7.1, pipeline-model-api:1.3.2, ace-editor:1.0.1, pipeline-model-declarative-agent:1.1.1, plain-credentials:1.3, workflow-cps-global-lib:2.11, pipeline-model-definition:1.3.2, workflow-job:2.25, workflow-basic-steps:2.11, pipeline-stage-step:2.3, pipeline-rest-api:2.10, jackson2-api:2.8.11.3, jquery-detached:1.2.1, cloudbees-folder:6.6, docker-workflow:1.14, docker-commons:1.5, workflow-support:2.20, workflow-multibranch:2.20, git:3.12.0, durable-task:1.26, workflow-cps:2.56, structs:1.19, credentials:2.1.18, git-client:2.7.7, workflow-api:2.29, job-dsl:1.76, ssh-credentials:1.13, pipeline-input-step:2.8, junit:1.3, workflow-step-api:2.20, mailer:1.20, display-url-api:2.0, junit:1.6, matrix-project:1.4.1, antisamy-markup-formatter:1.1, windows-slaves:1.0, matrix-auth:1.1, jsch:0.1.54.1, mailer:1.20, pipeline-stage-view:2.10, pipeline-graph-analysis:1.1, script-security:1.54, pipeline-build-step:2.7, credentials-binding:1.13, pipeline-stage-tags-metadata:1.3.2, display-url-api:1.0, workflow-scm-step:2.7, icon-shim:1.0.3, scm-api:2.6.3, momentjs:1.1, git-server:1.7, pipeline-model-extensions:1.3.2, pipeline-milestone-step:1.3.1, branch-api:2.0.18, authentication-tokens:1.1, handlebars:1.1, apache-httpcomponents-client-4-api:4.5.5-3.0, lockable-resources:2.3, workflow-aggregator:2.6] Remote Jenkins is version 2.190.2 Loading /Users/jenkins/Desktop/jenkins-test-harness/steps-2.176.3/src/main/dsl/TestJob.groovy Processing provided DSL script test-job (WorkflowJob): WOULD BE CREATEDdslUpdateJenkins results: WOULD BE CREATED: 1[Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@21f459fc{HTTP/1.1,[http/1.1]}{localhost:0} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Stopped scavenging [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.w.WebAppContext@93f432e{/jenkins,file:///Users/jenkins/Desktop/jenkins-test-harness/steps-2.176.3/build/jenkins-for-test/,UNAVAILABLE}{/Users/jenkins/Desktop/jenkins-test-harness/steps-2.176.3/build/jenkins-for-test}   When run on the 2.190.2 sample project, the following output is observed:   $ gradlew clean dslUpdateJenkins --dryRun --jenkinsUrl=https://my-production-jenkins.io/ --jenkinsUser=xxx --jenkinsApiToken=xxxxx > Task :dslUpdateJenkins WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-all/2.4.15/423a17aeb2f64bc6f76e8e44265a548bec80fd42/groovy-all-2.4.15.jar) to method java.lang.Object.finalize() WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.25/8dacf9514f0c707cbbcdd6fd699e8940d42fb54e/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-jdk14/1.7.26/c82cee90e4308cf6789d3d20255e30a670261fe7/slf4j-jdk14-1.7.26.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] [main] INFO org.eclipse.jetty.util.log - Logging initialized @716ms to org.eclipse.jetty.util.log.Slf4jLog Test timeout disabled. === Starting Run Job DSL Task [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - jetty-9.4.5.v20170502 [Executing Run Job DSL Task] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Scavenging every 600000ms [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@6ecab872{/jenkins,file:///Users/jenkins/Desktop/jenkins-test-harness/steps-2.190.2/build/jenkins-for-test/,AVAILABLE}{/Users/jenkins/Desktop/jenkins-test-harness/steps-2.190.2/build/jenkins-for-test} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@57402ba1{HTTP/1.1,[http/1.1]}{localhost:55171} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - Started @3000ms 4.892 [id=35] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Jenkins Git plugin v4.0.0 (git) java.io.IOException: Jenkins Git plugin version 4.0.0 failed to load. - SCM API Plugin version 2.4.1 is older than required. To fix, install version 2.6.3 or later. - Matrix Project Plugin version 1.4 is older than required. To fix, install version 1.14 or later. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:922) at hudson.PluginManager$2$1$1.run(PluginManager.java:545) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1118) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - jetty-9.4.5.v20170502 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Scavenging every 660000ms [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@5a50d9fc{/,null,AVAILABLE} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@dd71b20{HTTP/1.1,[http/1.1]}{0.0.0.0:55173} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.Server - Started @9306ms Installed plugins: [trilead-api:1.0.4, jdk-tool:1.0, command-launcher:1.2, script-security:1.56, jaxb:2.3.0, workflow-durable-task-step:2.22, git-client:3.0.0, pipeline-model-api:1.3.2, bouncycastle-api:2.16.0, ace-editor:1.0.1, credentials:2.3.0, pipeline-model-declarative-agent:1.1.1, plain-credentials:1.3, ssh-credentials:1.17.2, workflow-cps-global-lib:2.11, pipeline-model-definition:1.3.2, workflow-scm-step:2.9, workflow-job:2.25, workflow-basic-steps:2.11, pipeline-stage-step:2.3, pipeline-rest-api:2.10, apache-httpcomponents-client-4-api:4.5.10-1.0, jackson2-api:2.8.11.3, jquery-detached:1.2.1, cloudbees-folder:6.6, docker-workflow:1.14, docker-commons:1.5, workflow-support:2.20, workflow-multibranch:2.20, structs:1.20, durable-task:1.26, workflow-cps:2.56, workflow-api:2.29, mailer:1.23, job-dsl:1.76, junit:1.26.1, scm-api:2.4.1, workflow-step-api:2.20, structs:1.18, workflow-api:2.35, matrix-project:1.4, pipeline-input-step:2.8, junit:1.3, workflow-step-api:2.20, pipeline-stage-view:2.10, pipeline-graph-analysis:1.1, script-security:1.54, pipeline-build-step:2.7, credentials-binding:1.13, pipeline-stage-tags-metadata:1.3.2, display-url-api:1.0, jsch:0.1.55.1, mailer:1.20, display-url-api:2.0, matrix-auth:2.3, matrix-project:1.14, antisamy-markup-formatter:1.1, windows-slaves:1.0, icon-shim:1.0.3, scm-api:2.6.3, momentjs:1.1, git-server:1.7, pipeline-model-extensions:1.3.2, pipeline-milestone-step:1.3.1, branch-api:2.0.18, authentication-tokens:1.1, handlebars:1.1, lockable-resources:2.3, workflow-aggregator:2.6] Remote Jenkins is version 2.190.2 Loading /Users/jenkins/Desktop/jenkins-test-harness/steps-2.190.2/src/main/dsl/TestJob.groovy Processing provided DSL script test-job (WorkflowJob): WOULD BE CREATEDdslUpdateJenkins results: WOULD BE CREATED: 1[Executing Run Job DSL Task] INFO org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@57402ba1{HTTP/1.1,[http/1.1]}{localhost:0} [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.session - Stopped scavenging [Executing Run Job DSL Task] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.w.WebAppContext@6ecab872{/jenkins,file:///Users/jenkins/Desktop/jenkins-test-harness/steps-2.190.2/build/jenkins-for-test/,UNAVAILABLE}{/Users/jenkins/Desktop/jenkins-test-harness/steps-2.190.2/build/jenkins-for-test}   Note in the output for the second example, it complains that Git 4.0.0 cannot be installed:   SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Jenkins Git plugin v4.0.0 (git) java.io.IOException: Jenkins Git plugin version 4.0.0 failed to load. - SCM API Plugin version 2.4.1 is older than required. To fix, install version 2.6.3 or later. - Matrix Project Plugin version 1.4 is older than required. To fix, install version 1.14 or later. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:922) at hudson.PluginManager$2$1$1.run(PluginManager.java:545) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1118) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)   And when looking at the list of installed plugins further down, note how there is both an scm-api:2.4.1 and a scm-api:2.6.3 listed.   See jenkins-test-harness.zip for the three projects that can be used to reproduce this.   I was not entirely sure what project to file this issue with. While jenkins-war probably falls under core, there was an earlier ticket filed at https://issues.jenkins-ci.org/browse/JENKINS-59722 which got closed as being "not a bug" there. I am filing this under the jenkins-test-harness component now because based on my troubleshooting, I think it is behaving erroneously and should either do the right thing and not install 2 different versions of a plugin if an older one was found in the jenkins-war detached plugins, or allow people to exclude the loading of detached-plugins through some sort of flag. Another related ticket was found for https://github.com/heremaps/gradle-jenkins-jobdsl-plugin/issues/129 but given that the problem (and fix, or at least my local hack) all occur outside of the gradle-jenkins-jobdsl-plugin I am not convinced that's the right place to look for a fix.   Very curious what the core Jenkins maintainers think of this. I hope my sample projects (and hackish workaround) help illustrate better what we're running into here. Add Comment   This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.203288.1574816747000.7483.1574816760266%40Atlassian.JIRA.