Found this awesome plugin to solve the issue for monorepos https://github.com/TWChennai/gocd-git-path-material-plugin
On Thursday, September 8, 2016 at 10:15:45 AM UTC+5:30, Kaushik Chandrashekar wrote: > > Hi Varsha, > > I have a similar use case and would like to see this feature enhanced as > Brett mentioned :) > > We have Dockerfiles and the app in the same repo, so developers can use it > (this is not docker image of the app, but the software packages needed to > build the app). But we want to publish new docker images only when there is > a change to the Dockerfile and not after every commit to the app > > First pipeline is whitelisting the Dockerfile and the output (docker image > id) of which is consumed by the second pipeline which builds the app. But > since the second pipeline also has the repo as a material, the fan-in kicks > in and so changes to the app are not triggering any pipeline. I am waiting > for this feature which can be a workaround, > https://github.com/gocd/gocd/issues/437. But it would be nice if the > fan-in was smart enough to honor the blacklist/whitelist paths. > > Cheers, > Kaushik > > > On Monday, July 25, 2016 at 2:09:08 PM UTC+5:30, Brett Cave wrote: >> >> >> >> On Monday, 25 July 2016 10:13:12 UTC+2, Varsha Varadarajan wrote: >>> >>> Hey, >>> >>> I am trying to understand why you have set up the pipelines the way you >>> have. You have an upstream pipeline "BuildCore" and another pipeline >>> "BuildApp" dependent on 'BuildCore' and a git material 'repo' is common to >>> both. If your intention is to trigger the BuildCore for certain changes and >>> BuildApp for the rest of the changes in 'repo', can you please explain why >>> 'BuildApp' should be dependent on 'BuildCore'? >>> >> >> "repo/core" builds an artifact that is used by the application - a common >> library. This path is not updated that regularly. This library is also used >> by other pipelines. A new build of "Core" takes quite a long time.. >> "repo/app" builds an app that uses the core library. It is updated >> regularly, and builds relatively quickly. >> >> If a developer commits to "repo/app" on a daily basis the idea is to >> rebuild the app with the latest version of core - by this I mean compile >> the latest commit under "repo/app" path against the library built by the >> latest commit to "repo/core" path). Rebuilding "Core" for each "app" commit >> takes time and results in the same artifact being generated (as nothing in >> the core path has changed) - it slows down our CD process. I want to >> quickly build new revisions of app against 1 revision of core, unless core >> is updated. >> >> >>> >>> Thank you. >>> >>> On Monday, 25 July 2016 13:17:53 UTC+5:30, Brett Cave wrote: >>>> >>>> >>>> >>>> On Monday, 25 July 2016 08:50:07 UTC+2, Varsha Varadarajan wrote: >>>>> >>>>> The scenario: >>>>>> >>>>>> >>>>>> - A git repository "repo" with "core" and "app" root paths. >>>>>> - A "BuildCore" pipeline that uses "repo" with a whitelist on >>>>>> "core/*" - triggers automatically >>>>>> - A "BuildApp" pipeline that uses "repo" with a blacklist on >>>>>> "core/*" and an upstream pipeline dependency / material on >>>>>> "BuildCore" - >>>>>> triggers automatically >>>>>> >>>>>> The bug is that it seems like Go traces its upstream pipelines' >>>>>> materials and if a common material is found, the downstream pipeline >>>>>> will >>>>>> wait for the upstream pipeline to finish before triggering. >>>>>> >>>>>> >>>>>> >>>>> Given that the pipeline BuildApp is dependent on the git repository >>>>> 'repo' and pipeline 'BuildCore', any *valid* change in either >>>>> materials will trigger 'BuildApp'. In this scenario, a valid change is >>>>> any >>>>> change to 'core/*'. >>>>> If you observe the build cause for BuildApp, when a commit is made in >>>>> 'core/*', the build cause will only list the 'BuildCore' as the reason >>>>> for >>>>> the trigger. >>>>> It may seem that it is a violation of the blacklist in BuildApp, but a >>>>> change in core/* is not the direct reason for BuildApp being triggered. >>>>> It >>>>> is the expected behaviour. >>>>> >>>> >>>> I know it is the expected behaviour, but this behaviour could be >>>> improved to better suit build requirements. >>>> >>>> If I were to move the "core" and "app" paths to different repos, with >>>> "core" repo at revision "c01" and "app" repo at revision "a01", then I >>>> have >>>> a nice workflow in Go, the way it was designed: >>>> - If I commit "#c02" to the "core" repo, BuildCore will trigger. The >>>> new "#c02" build of BuildCore will trigger BuildApp with the old "#a01" >>>> revision from "app". >>>> - If I commit "#a02" to the "app" repo, the AppBuild pipeline will >>>> build "#a02" with "#c02". Subsequent commits to the app repo will build >>>> "#a03", "#a04", etc with "#c02" >>>> >>>> This workflow is not possible when using paths in 1 repo for different >>>> pipelines. I'm suggesting this is a shortfall in the workflow design - >>>> something that could be improved to give the same flow in a monolithic >>>> repo >>>> structure. The fan-in algorithm should not only trace materials, but also >>>> factor in black / whitelists as part of the algorithm. >>>> >>>> I am able to work around this and get the desired workflow by using a >>>> different hostname for the same repo in the 2 pipelines. >>>> >>>> >>>> >>>>> >>>>> In this scenario, if a commit is pushed to "app/foo", both BuildCore >>>>>> and BuildApp detect the new "repo" material. BuildCore doesn't trigger >>>>>> because only "core" is in its whitelist ("app/foo" is blacklisted). >>>>>> BuildApp finds the change, but doesn't trigger because it sees that >>>>>> BuildCore also uses the "repo" material and wants to wait for it to >>>>>> trigger >>>>>> and finish - it doesn't realize that BuildCore will not trigger due to >>>>>> the >>>>>> blacklist - BuildApp never triggers automatically. I'd imagine that it's >>>>>> only when I push to "core/bar" that BuildCore would trigger and that >>>>>> would >>>>>> then kick off BuildApp... >>>>>> >>>>> >>>>> Since BuildApp is dependent on BuildCore and git repository 'repo', >>>>> the fan-in algorithm kicks in where it tries to find a compatible >>>>> revision >>>>> with both the git material 'repo' and BuildCore. Since it cannot find >>>>> that >>>>> BuildCore has been run with "app/foo", BuildApp will never get triggered. >>>>> This behaviour exists irrespective of the blacklist and whitelist >>>>> feature. >>>>> And, because of the fan-in algorithm, the only valid change for BuildApp, >>>>> as mentioned above to be triggered is any change to 'core/*' only. >>>>> >>>>> >>>>>> >>>>>> I now either need to remove the black / white lists and build >>>>>> BuildCore for all commits or trigger manually. I'd imagine a solution to >>>>>> this is that BuildApp should not only check if "repo" is an upstream of >>>>>> "BuildCore" but should also ensure that the new "repo" material is not >>>>>> blacklisted there.... >>>>>> >>>>> >>>>> Yes, that is correct. >>>>> The whitelist on the upstream pipeline "BuildCore" is respected even >>>>> in BuildApp because of the dependency. A workaround would be to remove >>>>> the >>>>> whitelist in "BuildCore". However, keep in mind once the whitelist is >>>>> removed, the blacklist in downstream "BuildApp" will not be respected >>>>> (that >>>>> is, blacklist on "BuildApp" is useless in this scenario to have) as any >>>>> change triggering "BuildCore" will end up triggering "BuildApp" anyway. >>>>> >>>>> Thank You. >>>>> Varsha >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
