>From a user perspective, I like your idea of a "pipeline.sharedlibrary.test" step and/or a "run-pipeline" CLI command. They would definitely solve configuration issues with plugins, docker, etc. Now, Jesse mentioned some implementation challenges in his post from Wed, 07 Mar 2018 13:04:46 <https://groups.google.com/d/msg/jenkinsci-dev/gjz3CDhi-kk/yLIqqVGrAQAJ>, that I can't comment on, because I'm not that deep into Jenkins internals (yet?).
In the meantime, Jesse also provided a workaround for loading a "local" shared lib <https://stackoverflow.com/a/49112612/1845976>. With this I implemented my first (very simple) integration test <https://github.com/cloudogu/ces-build-lib/blob/cd08237a19380b52279f7bd596ab6ff492bea06a/test/it/com/cloudogu/ces/cesbuildlib/Sh/Jenkinsfile> and run it successfully from the Jenkinsfile of the shared lib <https://github.com/cloudogu/ces-build-lib/blob/feature/integration-test-POC/Jenkinsfile#L87> 🎉 You meant asserts should already work? How would you do those in that scenario? This approach has of course the limitations we talked about, regarding the Jenkins configuration . In addition these tests would only be run on Jenkins and not when running the build locally with Maven. So right now, I think a better approach would be to provide a Java library that allows for running Jenkinsfiles. It could be used to implement e.g. JUnit tests that are run by build tools such as maven locally (e.g. with the failsafe plugin) and in the same way in the Jenkinsfile. This library could abstract from the concrete execution with could either be - Jenkinsfile runner with local plugin folder - Jenkinsfile runner in Docker - "pipeline.sharedlibrary.test" step - a "run-pipeline" CLI command. The downside would be that such an implementation wouldn't be trivial. And, unfortunately, I can't invest to much of my free time right now. Am Dienstag, 6. März 2018 22:23:26 UTC+1 schrieb Kohsuke Kawaguchi: > > > > On Sun, Mar 4, 2018 at 8:13 AM <joha...@schnatterer.info <javascript:>> > wrote: > >> I think Jenkinsfile Runner brings a lot of opportunities for pipeline >> developers. The most obvious ones to me are >> >> 1. Pipeline development (Jenkinsfile) >> 2. Shared library development >> >> *Pipeline development* >> >> Right now (as described by others in this thread) pipeline development is >> either a loop of committing / fixing pipelines on production Jenkins, using >> pipeline replay on production Jenkins or setting up a local instance >> manually. >> >> With Jenkinsfile Runner we can get faster feedback without polluting our >> commit or Jenkins build history and don't have to set up a local instance >> manually. >> > Right. I think we all get this basic picture. Details are where things get > interesting! > > *Shared library development* >> >> Shared library development right now works much in the same as pipeline >> development, except that you have the library code and another (often >> production) Jenkinsfile to maintain, in order to try out (as opposed to >> automatically test) your Jenkinsfile. >> For shared libraries, we thankfully already have JenkinsPipelineUnit, >> that makes it easier to implement some tests. However, (as also mentioned >> by others in this thread) this is unit testing only. It mocks most of our >> environment. Often, green unit tests mean nothing for productive use of our >> share library. I even gave up test-driven development for shared libraries, >> in favor of 90s try-and-error-style programming. Because most of the time >> when trying the library with green unit tests in production, it turns out >> that the real Jenkins environment has some restriction that is beyond the >> scope of JenkinsPieplineUnit (e.g. sandboxing). >> > Worst thing about the current state is that we don't have reliable >> regression tests. A change in shared library with green unit tests is far >> from convincing me that the library will continue to work in production. >> >> With Jenkinsfile Runner we could write small Jenkinsfiles within the >> shared library repo's test folder and run them from the Jenkinsfile of the >> shared lib. Pretty much in the same way as we use Maven Invoker Plugin (as >> mentioned by Jesse) when developing maven plugins. >> > > OK, thank you, this is really helpful. I think maven invoker plugin > analogy is a great one for somebody like me. So from this perspective, > mocking isn't really high on the priority. > > >> >> *A first approach to shared library integration testing with Jenkinsfile >> Runner* >> My naive first approach was to build a Docker Image that contains >> Jenkinsfile Runner and all default plugins. >> >> docker run -v~/src/it/myfunction:/workspace schnatterer/jenkinsfile- >> runner:1.0-SNAPSHOT-2.108 /workspace >> >> runs the ~/foo/Jenkinsfile using Jenkinsfile Runner with all default >> plugins of Jenkins 2.108. >> >> My idea was to eventually do the same in Jenkinsfile of the shared lib >> like so (not tested) >> docker.image('schnatterer/jenkinsfile-runner:1.0-SNAPSHOT-2.108').inside >> { >> >> sh 'jenkinsfile-runner /app/jenkins /app/plugins src/it/myfunction' >> } >> or >> sh 'docker run --rm -v $(pwd):$(pwd) src/it/myfunction' >> > > So if this is the intended use case, then I have another idea. Instead of > running Jenkinsfile in this CLI and try to emulate your Jenkins instance as > closely as possible (in terms of configuration), we could just run > Jenkinsfile in the current Jenkins, in a place that nobody else can see. > > For example, packaged as CLI, you'd run it like: > > $ java -jar jenkins-cli.jar -s https://jenkins.example.com/ > run-pipeline ./src/it/myfunction > > Or from within Jenkinsfile, could be something like: > > pipeline.sharedlibrary.test './src/it/myfunction' > > ... and this would completely bypass the challenge of trying to replicate > the Jenkins configuration. > > > >> It turned out that there a two major problems: >> >> 1. There's no way to add non-default Jenkins plugins. >> My local test for cloudogu/ces-build-lib >> <https://github.com/cloudogu/ces-build-lib> failed because there was >> no GitHub Groovy Libraries plugin. >> Here, my hope is that Configuration-as-code plugin might help >> automate loading plugins. >> >> Yeah, I'm really looking forward for CaC project to fill this gap, and I > know they are working on it. > <https://github.com/jenkinsci/jep/pull/59/files> > >> >> 1. There's still no way to load a "local" shared library from the >> same repo. So, we still would have to find a way to configure the shared >> library in our Jenkinsfile Runner. >> Loading local shared libraries has already been discussed here >> <https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37> >> and there <https://stackoverflow.com/q/46213913/1845976>. >> >> Good point. > > >> Once those issues are solved, we'll have a very basic way of automating >> integration tests for shared libraries by executing IT Jenkinsfiles from >> the shared libraries pipeline and failing the build if the IT fails. >> > > That'd be really cool, isn't it!? > >> Of course, this would be very basic testing. For more sophistiated >> testing we would want to >> >> >> - trigger the ITs from maven or gradle, >> - use asserts, >> - get the results as JUnit XML. >> >> So, yes, we're not there yet. But we now have a foundation to build all >> this upon. >> > > asserts would already work, right? It seems like we can do that relatively > easily on its own. > > But with that aside, I think I understand the picture in your head. You > are really approaching it like Maven plugin development. Indeed this would > help illustrate relative strength of both JenkinsPipelineUnit and > "integration tests" > > >> >> Thanks for that & best regards, >> >> Johannes >> >> >> Am Donnerstag, 1. März 2018 20:23:15 UTC+1 schrieb Kohsuke Kawaguchi: >>> >>> Jenkinsfile Runner is an experiment to package Jenkins pipeline >>> execution as a command line tool. The intend use cases include: >>> >>> - Use Jenkins in Function-as-a-Service context >>> - Assist editing Jenkinsfile locally >>> - Integration test shared libraries >>> >>> Over the past year, I've done some deep-dive 1:1 conversations with some >>> Jenkins users and I felt something like this might move the needle for them >>> in an important way. >>> >>> I'd love to hear any reactions on your side. Could something like this >>> be important for you, does it miss any key points for you? If you mentally >>> picture a perfect version of this, what would that do, and how would you >>> use? >>> >>> Let me know! >>> >>> -- >>> Kohsuke Kawaguchi >>> >> >> >> Am Donnerstag, 1. März 2018 20:23:15 UTC+1 schrieb Kohsuke Kawaguchi: >>> >>> Jenkinsfile Runner is an experiment to package Jenkins pipeline >>> execution as a command line tool. The intend use cases include: >>> >>> - Use Jenkins in Function-as-a-Service context >>> - Assist editing Jenkinsfile locally >>> - Integration test shared libraries >>> >>> Over the past year, I've done some deep-dive 1:1 conversations with some >>> Jenkins users and I felt something like this might move the needle for them >>> in an important way. >>> >>> I'd love to hear any reactions on your side. Could something like this >>> be important for you, does it miss any key points for you? If you mentally >>> picture a perfect version of this, what would that do, and how would you >>> use? >>> >>> Let me know! >>> >>> -- >>> Kohsuke Kawaguchi >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-de...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/fcdcb7e4-f6ca-4ec5-bf7d-0c9ca32618ee%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-dev/fcdcb7e4-f6ca-4ec5-bf7d-0c9ca32618ee%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > Kohsuke Kawaguchi > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/25a576e2-8c2b-4da4-b5c1-568b632738d8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
