So sometimes with one repo and multiple pipelines you want fan-in behaviour, e.g if you are splitting the pipelines to run different logical testing blocks in parallel and either
- don't want to model them as parallel jobs within one pipeline or - where pipelines require multiple source code material inputs, and you don't want excessive triggering of unrelated pipeline work Basically the classic cases documented at https://www.gocd.org/getting-started/part-3/#fan_out_fan_in or https://www.gocd.org/2017/04/17/build-propagation-using-fan-in-fan-out.html Other times you have multiple things in one repo for ease of management, but things are sufficiently segregated and you are OK with treating the pieces independently. In this case fan-in might not be what you want/need, despite sharing a repository/material. Let's say you have some more classic mono repo A: ./deploy-scripts/ ./service-common/ ./service-A/ ./frontend/ Perhaps you then have pipelines like *repoA *--*fanout*--> frontend-build --> frontend-functional-test --*fanin*--> frontend-plus-service-A-smoke-test --*fanout*--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT) *repoA* --*fanout*--> service-A-build --> service-A-functional-test -- *fanin*--> frontend-plus-service-A-smoke-test --*fanout*--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT) In a normal setup with GoCD, frontend-plus-service-A-smoke-test will only trigger when all of the upstreams are built on the same revision from repoA. If the frontend tests are perhaps far slower, this may unnecessarily slow progress down for promoting service-A code changes to UAT, and require triggering of everything. You may be confident enough that they can be built and deployed independently, but still want some confidence that the frontend and service-A play nicely together which is why you have "frontend-plus-service-A-smoke-test". In this case you may want to take more responsibility on yourselves, and consider something like https://github.com/TWChennai/gocd-git-path-material-plugin to allow the deploys to happen independently, but without excessive triggering of unrelated pieces, without splitting your repository. With something like this you might instead model like *repoA (path: deploy-scripts/ + frontend/*) --> frontend-build --> frontend-functional-test --> frontend-plus-service-A-smoke-test --*fanout*--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT) *repoA (path: deploy-scripts/ + service-common/ + service-A/)* --> service-A-build --> service-A-functional-test --> frontend-plus-service-A-smoke-test --*fanout*--> (deploy-frontend-to-UAT + deploy-service-A-to-UAT) This would logically split the materials while allowing you to commit together in one repo, essentially working around the limitations of filtering noted here <https://docs.gocd.org/current/advanced_usage/fan_in.html#limitations>. Anyway, these are just different tools in the arsenal, depending on what you are trying to achieve, and how you are modelling your pipelines and splitting your source code, test/deploy scripting etc. -Chad On Tue, Jun 13, 2023 at 8:39 PM Manikkumar K <[email protected]> wrote: > You are spot on. We do share one of the repositories for the all > microservice(aka upstreams). > And was able to simulate and see where the fan-in didn't happen when > upstreams do not share any of the materials. > > Thanks for prompt response, > Mani > > On Tue, Jun 13, 2023 at 8:43 AM Chad Wilson <[email protected]> > wrote: > >> If it's "not monorepo", can you clarify if you have multiple upstreams >> that point to one repository/material? That is the scenario that triggers >> fan-in, and that does at least indicate that the repo is somewhat being >> treated as a monorepo in the sense that you have multiple pipelines >> directly dependent on that repository. >> >> Otherwise you might need to share more specific examples of pipelines, >> dependencies and the materials they have, along with the behaviour you >> observe vs. desire. >> >> To successfully trigger a downstream, all upstreams only need to pass (on >> a given material version) if they share a material. If they *do not* >> share 1 or more materials then the behaviour should be as you describe - >> the downstream will trigger with the last successful version of each >> upstream stage (no fan-in required). >> >> -Chad >> >> On Tue, Jun 13, 2023 at 11:01 AM Manikkumar K < >> [email protected]> wrote: >> >>> Ours is not monorepo. Is there any other alternative? >>> >>> On Tue, Jun 13, 2023, 07:18 Chad Wilson <[email protected]> wrote: >>> >>>> Doing that means removing the guarantees fan-in offers you as to >>>> consistency of the material versions on upstream pipelines when triggering >>>> a downstream. >>>> >>>> It's possible to turn off fan-in globally (although I have never done >>>> so personally) but to my knowledge not for individual pipelines. >>>> >>>> Alternatively, if your upstreams are some sort of monorepo which are >>>> segregated by upstream pipelines (e.g each pipeline operates on different >>>> subdirectories or parts of the repo) you could consider using something >>>> like https://github.com/TWChennai/gocd-git-path-material-plugin to >>>> define separate materials for each of the upstreams that share a repo with >>>> different subsets of the repo that they operate on so GoCD doesn't decide >>>> they need fan-in. >>>> >>>> -Chad >>>> >>>> On Tue, 13 Jun 2023, 08:39 Manikkumar K, <[email protected]> >>>> wrote: >>>> >>>>> We do have a fan-in where number upstream has exploded in the count, >>>>> >>>>> Instead of waiting for all upstream to pass, can we pick most recent >>>>> successful upstream pipeline along with new pipeline instance of one(or >>>>> more) of upstream? >>>>> >>>>> Thanks, >>>>> Mani >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/4bcad889-0820-427b-ac85-d91e50f24e43n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/go-cd/4bcad889-0820-427b-ac85-d91e50f24e43n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/VVTYzU5OhV4/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-Pub%3DUtDr0QEc-yS5DoR3BbKbagC0Jk8rfSYFDbiXe5Q%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-Pub%3DUtDr0QEc-yS5DoR3BbKbagC0Jk8rfSYFDbiXe5Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CACfO_cKcqovDwHrmD0G_yBP1cCYJJT%2BMs5hxV9KeTKrQmuf-OQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CACfO_cKcqovDwHrmD0G_yBP1cCYJJT%2BMs5hxV9KeTKrQmuf-OQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "go-cd" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/go-cd/VVTYzU5OhV4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAA1RwH8n%2BOm%3D5L13B3srVAhNZAPnHdrPZKJMh2sC7ZQrCB%2BJBw%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8n%2BOm%3D5L13B3srVAhNZAPnHdrPZKJMh2sC7ZQrCB%2BJBw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CACfO_c%2BrsZviV4sMiy6%2BnTquWqWi7L6AafDc3%2B5Z%2BbhCrihzjg%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CACfO_c%2BrsZviV4sMiy6%2BnTquWqWi7L6AafDc3%2B5Z%2BbhCrihzjg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8xfENOsjrxjLOwFB5ALh4uYN1Ygarb99DC94W9%2By2B_Q%40mail.gmail.com.
