My team faces similar challenges, and I agree with almost everything said so far. I definitely echo the sentiment that it feels like there is a lot of wasted repetition when invoking discrete phases separately. We currently do this via separate jobs now, which often run at different frequencies. However, we have been able to configure Maven such that we really only have two repetitious phases, across these multiple jobs, which luckily are not costly for us. Maven generate-resources and compile phases, as we can skip over tests and deployments which are not part of the logical unit being executed. In the end, even though we know we are wasting some execution time, it turns out not to be all that much at all.. and isn't yet worthy of any kind of optimization or special attention.
So right now our (~2min) compile+unit test runs every commit and only runs Junit tests. Our (~8min) compile+integration test runs almost every commit+jenkins build window to group multiple commits and only runs Arquillian tests. Our (~15min) compile+user acceptance test runs every day and only runs Selenium tests. Then we have lots of on the side jobs which ensure legal compliance, code coverage, file encoding fixes, code style fixes, static analysis, etc etc, which run hourly, daily, or weekly, and are mostly fast (<5min). We are exploring Jenkins Pipeline in order to chain, both sequential where necessary and then parallel where it makes sense, these discrete jobs in such a way that we will have a continuous delivery available at the end which has run the full gauntlet of all our testing and best practices verification for every commit without doing unnecessary and expensive steps, failing the pipeline as soon as possible. Each commit triggers a build which triggers a new git branch which is passed to subsequent phases, if any stage fails, the branch is deleted and makes no further progress. Additionally, we are trying to leverage separate Jenkins jobs for each discrete job so that we can trigger it on demand in order to do some quick checking, we also find that we can get easier cross-project reuse using that technique. We're using Git, Apache Maven, Jenkins CI, Jacoco, Arquillian, SonarQube, Checkstyle, PMD, FindBugs, and many others. There are only two critiques we can find with this style: 1) the aforementioned modest amount of wasted generate-resources and compile phases across multiple jobs in the same pipeline, and 2) each phase is producing artifacts to be tested and not reusing the ones built and verified at previous stages, thus there may be a hypothetical difference between what different stages are actually testing, though we think the risk of this is not consequential. This technique otherwise seems like a holy grail to me and the team. I welcome ideas about it. On Mon, Jul 25, 2016 at 10:36 AM, Victor Noël <[email protected]> wrote: > Thanks for your feedbacks! > > So in the end, what is the solution you adopted? more coarse grained stages? > > Le dimanche 24 juillet 2016 14:18:18 UTC+2, [email protected] a écrit : >> >> Hello Victor, >> >> I am faced with a similar problem and gave up on the idea rather early on. >> Here are some of the characteristics of multiple stages of a pipeline >> >> Ability of each stage to be explictly invoked by some command >> Ability of each stage to be re-run in such a way that it can start off >> from where the previous stage finished >> >> The maven lifecycle reactor does not play well with these concepts. For >> example, there is no elegant way to tell maven to compile the code assuming >> that the generation phase has happened. Likewise, one cannot ask maven to >> simply execute the tests assuming that compilation has already taken place. >> The short answer that the benefits of breaking something into various >> pipeline stages is lost when maven. >> >> The situation with gradle is not better as far as the relationship with >> the build tool and jenkins is concerned. However, the fact that gradle is >> much better at quickly checking if earlier steps have been completed makes >> it only marginally better. >> >> >> On Saturday, July 23, 2016 at 1:10:08 PM UTC+5:30, Victor Noël wrote: >>> >>> Hi, >>> >>> I'm trying to wrap wy head around moving to pipeline jobs. >>> >>> Currently I use maven jobs (but my question applies to freestyle jobs >>> that execute maven) by executing one single build command that will validate >>> constraints (checkstyle and/or other stuffs like that), generate code, >>> compile, run test, run it test and deploy artefacts. >>> >>> What is the best way to convert to pipeline with multiple stages? >>> >>> My main concern is that if I simply run each of the step with maven in >>> each stage, then I feel like some of steps from maven may be executed >>> multiple times (such as test being executed again for it tests or for >>> deploy). >>> >>> Is there some success stories around here on how to handle such a >>> migration? >>> >>> Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/e869c7c4-2ea5-4d8b-8268-7a083348b2b7%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAArU9ibW11LmyO7TSC%2By7K8rynfngHXOMy9Enuh2Stanzx1wxg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
