On Sun, Mar 11, 2018 at 6:43 AM <[email protected]> wrote: > From a user perspective, I like your idea of a > "pipeline.sharedlibrary.test" step and/or a "run-pipeline" CLI command. > They would definitely solve configuration issues with plugins, docker, etc. > Now, Jesse mentioned some implementation challenges in his post from Wed, > 07 Mar 2018 13:04:46 > <https://groups.google.com/d/msg/jenkinsci-dev/gjz3CDhi-kk/yLIqqVGrAQAJ>, > that I can't comment on, because I'm not that deep into Jenkins internals > (yet?). > > In the meantime, Jesse also provided a workaround for loading a "local" > shared lib <https://stackoverflow.com/a/49112612/1845976>. With this I > implemented my first (very simple) integration test > <https://github.com/cloudogu/ces-build-lib/blob/cd08237a19380b52279f7bd596ab6ff492bea06a/test/it/com/cloudogu/ces/cesbuildlib/Sh/Jenkinsfile> > and run it successfully from the Jenkinsfile of the shared lib > <https://github.com/cloudogu/ces-build-lib/blob/feature/integration-test-POC/Jenkinsfile#L87> > 🎉 >
Nice! You meant asserts should already work? How would you do those in that > scenario? > I was thinking you'd just put assert statements into Jenkinsfile and that should already work. Granted, I haven't tried it so maybe it doesn't work. This approach has of course the limitations we talked about, regarding the > Jenkins configuration . > In addition these tests would only be run on Jenkins and not when running > the build locally with Maven. > I'm not sure if I follow you here. The two approaches that I was thinking about were: - "Jenkinsfile Runner" that runs embedded Jenkins locally and runs Jenkinsfile in there - "run-pipeline CLI command" that submits Jenkinsfile into an existing Jenkins server and run it there and send the result back "pipeline.sharedlibrary.test" is just a pipeline step that runs tests in some form. In each of the above two versions we can think of their corresponding "pipeline.sharedlibrary.test" step. > So right now, I think a better approach would be to provide a Java library > that allows for running Jenkinsfiles. > It could be used to implement e.g. JUnit tests that are run by build tools > such as maven locally (e.g. with the failsafe plugin) and in the same way > in the Jenkinsfile. > This library could abstract from the concrete execution with could either > be > - Jenkinsfile runner with local plugin folder > - Jenkinsfile runner in Docker > - "pipeline.sharedlibrary.test" step > - a "run-pipeline" CLI command. > I was kind of trying to avoid doing 4 different things that each has different unique trade-offs that only Jenkins experts would understand. You have mentioned two primary use cases for you, pipeline development and shared library development. I think we can find one solution that works for those two cases. I just need to think about this. The downside would be that such an implementation wouldn't be trivial. And, > unfortunately, I can't invest to much of my free time right now. > Your feedback has been very helpful, and I appreciate the time you are investing in those. > > Am Dienstag, 6. März 2018 22:23:26 UTC+1 schrieb Kohsuke Kawaguchi: >> >> >> >> On Sun, Mar 4, 2018 at 8:13 AM <[email protected]> wrote: >> >>> I think Jenkinsfile Runner brings a lot of opportunities for pipeline >>> developers. The most obvious ones to me are >>> >>> 1. Pipeline development (Jenkinsfile) >>> 2. Shared library development >>> >>> *Pipeline development* >>> >>> Right now (as described by others in this thread) pipeline development >>> is either a loop of committing / fixing pipelines on production Jenkins, >>> using pipeline replay on production Jenkins or setting up a local instance >>> manually. >>> >>> With Jenkinsfile Runner we can get faster feedback without polluting our >>> commit or Jenkins build history and don't have to set up a local instance >>> manually. >>> >> Right. I think we all get this basic picture. Details are where things >> get interesting! >> >> *Shared library development* >>> >>> Shared library development right now works much in the same as pipeline >>> development, except that you have the library code and another (often >>> production) Jenkinsfile to maintain, in order to try out (as opposed to >>> automatically test) your Jenkinsfile. >>> For shared libraries, we thankfully already have JenkinsPipelineUnit, >>> that makes it easier to implement some tests. However, (as also mentioned >>> by others in this thread) this is unit testing only. It mocks most of our >>> environment. Often, green unit tests mean nothing for productive use of our >>> share library. I even gave up test-driven development for shared libraries, >>> in favor of 90s try-and-error-style programming. Because most of the time >>> when trying the library with green unit tests in production, it turns out >>> that the real Jenkins environment has some restriction that is beyond the >>> scope of JenkinsPieplineUnit (e.g. sandboxing). >>> >> Worst thing about the current state is that we don't have reliable >>> regression tests. A change in shared library with green unit tests is far >>> from convincing me that the library will continue to work in production. >>> >>> With Jenkinsfile Runner we could write small Jenkinsfiles within the >>> shared library repo's test folder and run them from the Jenkinsfile of the >>> shared lib. Pretty much in the same way as we use Maven Invoker Plugin (as >>> mentioned by Jesse) when developing maven plugins. >>> >> >> OK, thank you, this is really helpful. I think maven invoker plugin >> analogy is a great one for somebody like me. So from this perspective, >> mocking isn't really high on the priority. >> >> >>> >>> *A first approach to shared library integration testing with Jenkinsfile >>> Runner* >>> My naive first approach was to build a Docker Image that contains >>> Jenkinsfile Runner and all default plugins. >>> >>> docker run -v~/src/it/myfunction:/workspace schnatterer/jenkinsfile- >>> runner:1.0-SNAPSHOT-2.108 /workspace >>> >>> runs the ~/foo/Jenkinsfile using Jenkinsfile Runner with all default >>> plugins of Jenkins 2.108. >>> >>> My idea was to eventually do the same in Jenkinsfile of the shared lib >>> like so (not tested) >>> docker.image('schnatterer/jenkinsfile-runner:1.0-SNAPSHOT-2.108').inside >>> { >>> >>> sh 'jenkinsfile-runner /app/jenkins /app/plugins src/it/myfunction' >>> } >>> or >>> sh 'docker run --rm -v $(pwd):$(pwd) src/it/myfunction' >>> >> >> So if this is the intended use case, then I have another idea. Instead of >> running Jenkinsfile in this CLI and try to emulate your Jenkins instance as >> closely as possible (in terms of configuration), we could just run >> Jenkinsfile in the current Jenkins, in a place that nobody else can see. >> >> For example, packaged as CLI, you'd run it like: >> >> $ java -jar jenkins-cli.jar -s https://jenkins.example.com/ >> run-pipeline ./src/it/myfunction >> >> Or from within Jenkinsfile, could be something like: >> >> pipeline.sharedlibrary.test './src/it/myfunction' >> >> ... and this would completely bypass the challenge of trying to replicate >> the Jenkins configuration. >> >> >> >>> It turned out that there a two major problems: >>> >>> 1. There's no way to add non-default Jenkins plugins. >>> My local test for cloudogu/ces-build-lib >>> <https://github.com/cloudogu/ces-build-lib> failed because there was >>> no GitHub Groovy Libraries plugin. >>> Here, my hope is that Configuration-as-code plugin might help >>> automate loading plugins. >>> >>> Yeah, I'm really looking forward for CaC project to fill this gap, and I >> know they are working on it. >> <https://github.com/jenkinsci/jep/pull/59/files> >> >>> >>> 1. There's still no way to load a "local" shared library from the >>> same repo. So, we still would have to find a way to configure the shared >>> library in our Jenkinsfile Runner. >>> Loading local shared libraries has already been discussed here >>> <https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37> >>> and there <https://stackoverflow.com/q/46213913/1845976>. >>> >>> Good point. >> >> >>> Once those issues are solved, we'll have a very basic way of automating >>> integration tests for shared libraries by executing IT Jenkinsfiles from >>> the shared libraries pipeline and failing the build if the IT fails. >>> >> >> That'd be really cool, isn't it!? >> >>> Of course, this would be very basic testing. For more sophistiated >>> testing we would want to >>> >>> >>> - trigger the ITs from maven or gradle, >>> - use asserts, >>> - get the results as JUnit XML. >>> >>> So, yes, we're not there yet. But we now have a foundation to build all >>> this upon. >>> >> >> asserts would already work, right? It seems like we can do that >> relatively easily on its own. >> >> But with that aside, I think I understand the picture in your head. You >> are really approaching it like Maven plugin development. Indeed this would >> help illustrate relative strength of both JenkinsPipelineUnit and >> "integration tests" >> >> > >>> Thanks for that & best regards, >>> >>> Johannes >>> >>> >>> Am Donnerstag, 1. März 2018 20:23:15 UTC+1 schrieb Kohsuke Kawaguchi: >>>> >>>> Jenkinsfile Runner is an experiment to package Jenkins pipeline >>>> execution as a command line tool. The intend use cases include: >>>> >>>> - Use Jenkins in Function-as-a-Service context >>>> - Assist editing Jenkinsfile locally >>>> - Integration test shared libraries >>>> >>>> Over the past year, I've done some deep-dive 1:1 conversations with >>>> some Jenkins users and I felt something like this might move the needle for >>>> them in an important way. >>>> >>>> I'd love to hear any reactions on your side. Could something like this >>>> be important for you, does it miss any key points for you? If you mentally >>>> picture a perfect version of this, what would that do, and how would you >>>> use? >>>> >>>> Let me know! >>>> >>>> -- >>>> Kohsuke Kawaguchi >>>> >>> >>> >>> Am Donnerstag, 1. März 2018 20:23:15 UTC+1 schrieb Kohsuke Kawaguchi: >>>> >>>> Jenkinsfile Runner is an experiment to package Jenkins pipeline >>>> execution as a command line tool. The intend use cases include: >>>> >>>> - Use Jenkins in Function-as-a-Service context >>>> - Assist editing Jenkinsfile locally >>>> - Integration test shared libraries >>>> >>>> Over the past year, I've done some deep-dive 1:1 conversations with >>>> some Jenkins users and I felt something like this might move the needle for >>>> them in an important way. >>>> >>>> I'd love to hear any reactions on your side. Could something like this >>>> be important for you, does it miss any key points for you? If you mentally >>>> picture a perfect version of this, what would that do, and how would you >>>> use? >>>> >>>> Let me know! >>>> >>>> -- >>>> Kohsuke Kawaguchi >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> >> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >> >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/fcdcb7e4-f6ca-4ec5-bf7d-0c9ca32618ee%40googlegroups.com >>> <https://groups.google.com/d/msgid/jenkinsci-dev/fcdcb7e4-f6ca-4ec5-bf7d-0c9ca32618ee%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> Kohsuke Kawaguchi >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/25a576e2-8c2b-4da4-b5c1-568b632738d8%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/25a576e2-8c2b-4da4-b5c1-568b632738d8%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Kohsuke Kawaguchi -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4wdH2usB_k8GtOOb5chmnTuTf93PM3YeqZcgXnfOaqqEQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
