Hi Varsha, I have a similar use case and would like to see this feature enhanced as Brett mentioned :)
We have Dockerfiles and the app in the same repo, so developers can use it (this is not docker image of the app, but the software packages needed to build the app). But we want to publish new docker images only when there is a change to the Dockerfile and not after every commit to the app First pipeline is whitelisting the Dockerfile and the output (docker image id) of which is consumed by the second pipeline which builds the app. But since the second pipeline also has the repo as a material, the fan-in kicks in and so changes to the app are not triggering any pipeline. I am waiting for this feature which can be a workaround, https://github.com/gocd/gocd/issues/437. But it would be nice if the fan-in was smart enough to honor the blacklist/whitelist paths. Cheers, Kaushik On Monday, July 25, 2016 at 2:09:08 PM UTC+5:30, Brett Cave wrote: > > > > On Monday, 25 July 2016 10:13:12 UTC+2, Varsha Varadarajan wrote: >> >> Hey, >> >> I am trying to understand why you have set up the pipelines the way you >> have. You have an upstream pipeline "BuildCore" and another pipeline >> "BuildApp" dependent on 'BuildCore' and a git material 'repo' is common to >> both. If your intention is to trigger the BuildCore for certain changes and >> BuildApp for the rest of the changes in 'repo', can you please explain why >> 'BuildApp' should be dependent on 'BuildCore'? >> > > "repo/core" builds an artifact that is used by the application - a common > library. This path is not updated that regularly. This library is also used > by other pipelines. A new build of "Core" takes quite a long time.. > "repo/app" builds an app that uses the core library. It is updated > regularly, and builds relatively quickly. > > If a developer commits to "repo/app" on a daily basis the idea is to > rebuild the app with the latest version of core - by this I mean compile > the latest commit under "repo/app" path against the library built by the > latest commit to "repo/core" path). Rebuilding "Core" for each "app" commit > takes time and results in the same artifact being generated (as nothing in > the core path has changed) - it slows down our CD process. I want to > quickly build new revisions of app against 1 revision of core, unless core > is updated. > > >> >> Thank you. >> >> On Monday, 25 July 2016 13:17:53 UTC+5:30, Brett Cave wrote: >>> >>> >>> >>> On Monday, 25 July 2016 08:50:07 UTC+2, Varsha Varadarajan wrote: >>>> >>>> The scenario: >>>>> >>>>> >>>>> - A git repository "repo" with "core" and "app" root paths. >>>>> - A "BuildCore" pipeline that uses "repo" with a whitelist on >>>>> "core/*" - triggers automatically >>>>> - A "BuildApp" pipeline that uses "repo" with a blacklist on >>>>> "core/*" and an upstream pipeline dependency / material on "BuildCore" >>>>> - >>>>> triggers automatically >>>>> >>>>> The bug is that it seems like Go traces its upstream pipelines' >>>>> materials and if a common material is found, the downstream pipeline will >>>>> wait for the upstream pipeline to finish before triggering. >>>>> >>>>> >>>>> >>>> Given that the pipeline BuildApp is dependent on the git repository >>>> 'repo' and pipeline 'BuildCore', any *valid* change in either >>>> materials will trigger 'BuildApp'. In this scenario, a valid change is any >>>> change to 'core/*'. >>>> If you observe the build cause for BuildApp, when a commit is made in >>>> 'core/*', the build cause will only list the 'BuildCore' as the reason for >>>> the trigger. >>>> It may seem that it is a violation of the blacklist in BuildApp, but a >>>> change in core/* is not the direct reason for BuildApp being triggered. It >>>> is the expected behaviour. >>>> >>> >>> I know it is the expected behaviour, but this behaviour could be >>> improved to better suit build requirements. >>> >>> If I were to move the "core" and "app" paths to different repos, with >>> "core" repo at revision "c01" and "app" repo at revision "a01", then I have >>> a nice workflow in Go, the way it was designed: >>> - If I commit "#c02" to the "core" repo, BuildCore will trigger. The >>> new "#c02" build of BuildCore will trigger BuildApp with the old "#a01" >>> revision from "app". >>> - If I commit "#a02" to the "app" repo, the AppBuild pipeline will >>> build "#a02" with "#c02". Subsequent commits to the app repo will build >>> "#a03", "#a04", etc with "#c02" >>> >>> This workflow is not possible when using paths in 1 repo for different >>> pipelines. I'm suggesting this is a shortfall in the workflow design - >>> something that could be improved to give the same flow in a monolithic repo >>> structure. The fan-in algorithm should not only trace materials, but also >>> factor in black / whitelists as part of the algorithm. >>> >>> I am able to work around this and get the desired workflow by using a >>> different hostname for the same repo in the 2 pipelines. >>> >>> >>> >>>> >>>> In this scenario, if a commit is pushed to "app/foo", both BuildCore >>>>> and BuildApp detect the new "repo" material. BuildCore doesn't trigger >>>>> because only "core" is in its whitelist ("app/foo" is blacklisted). >>>>> BuildApp finds the change, but doesn't trigger because it sees that >>>>> BuildCore also uses the "repo" material and wants to wait for it to >>>>> trigger >>>>> and finish - it doesn't realize that BuildCore will not trigger due to >>>>> the >>>>> blacklist - BuildApp never triggers automatically. I'd imagine that it's >>>>> only when I push to "core/bar" that BuildCore would trigger and that >>>>> would >>>>> then kick off BuildApp... >>>>> >>>> >>>> Since BuildApp is dependent on BuildCore and git repository 'repo', the >>>> fan-in algorithm kicks in where it tries to find a compatible revision >>>> with >>>> both the git material 'repo' and BuildCore. Since it cannot find that >>>> BuildCore has been run with "app/foo", BuildApp will never get triggered. >>>> This behaviour exists irrespective of the blacklist and whitelist feature. >>>> And, because of the fan-in algorithm, the only valid change for BuildApp, >>>> as mentioned above to be triggered is any change to 'core/*' only. >>>> >>>> >>>>> >>>>> I now either need to remove the black / white lists and build >>>>> BuildCore for all commits or trigger manually. I'd imagine a solution to >>>>> this is that BuildApp should not only check if "repo" is an upstream of >>>>> "BuildCore" but should also ensure that the new "repo" material is not >>>>> blacklisted there.... >>>>> >>>> >>>> Yes, that is correct. >>>> The whitelist on the upstream pipeline "BuildCore" is respected even in >>>> BuildApp because of the dependency. A workaround would be to remove the >>>> whitelist in "BuildCore". However, keep in mind once the whitelist is >>>> removed, the blacklist in downstream "BuildApp" will not be respected >>>> (that >>>> is, blacklist on "BuildApp" is useless in this scenario to have) as any >>>> change triggering "BuildCore" will end up triggering "BuildApp" anyway. >>>> >>>> Thank You. >>>> Varsha >>>> >>> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
