On Monday, 25 July 2016 10:13:12 UTC+2, Varsha Varadarajan wrote:
>
> Hey,
>
> I am trying to understand why you have set up the pipelines the way you
> have. You have an upstream pipeline "BuildCore" and another pipeline
> "BuildApp" dependent on 'BuildCore' and a git material 'repo' is common to
> both. If your intention is to trigger the BuildCore for certain changes and
> BuildApp for the rest of the changes in 'repo', can you please explain why
> 'BuildApp' should be dependent on 'BuildCore'?
>
"repo/core" builds an artifact that is used by the application - a common
library. This path is not updated that regularly. This library is also used
by other pipelines. A new build of "Core" takes quite a long time..
"repo/app" builds an app that uses the core library. It is updated
regularly, and builds relatively quickly.
If a developer commits to "repo/app" on a daily basis the idea is to
rebuild the app with the latest version of core - by this I mean compile
the latest commit under "repo/app" path against the library built by the
latest commit to "repo/core" path). Rebuilding "Core" for each "app" commit
takes time and results in the same artifact being generated (as nothing in
the core path has changed) - it slows down our CD process. I want to
quickly build new revisions of app against 1 revision of core, unless core
is updated.
>
> Thank you.
>
> On Monday, 25 July 2016 13:17:53 UTC+5:30, Brett Cave wrote:
>>
>>
>>
>> On Monday, 25 July 2016 08:50:07 UTC+2, Varsha Varadarajan wrote:
>>>
>>> The scenario:
>>>>
>>>>
>>>> - A git repository "repo" with "core" and "app" root paths.
>>>> - A "BuildCore" pipeline that uses "repo" with a whitelist on
>>>> "core/*" - triggers automatically
>>>> - A "BuildApp" pipeline that uses "repo" with a blacklist on
>>>> "core/*" and an upstream pipeline dependency / material on "BuildCore"
>>>> -
>>>> triggers automatically
>>>>
>>>> The bug is that it seems like Go traces its upstream pipelines'
>>>> materials and if a common material is found, the downstream pipeline will
>>>> wait for the upstream pipeline to finish before triggering.
>>>>
>>>>
>>>>
>>> Given that the pipeline BuildApp is dependent on the git repository
>>> 'repo' and pipeline 'BuildCore', any *valid* change in either materials
>>> will trigger 'BuildApp'. In this scenario, a valid change is any change to
>>> 'core/*'.
>>> If you observe the build cause for BuildApp, when a commit is made in
>>> 'core/*', the build cause will only list the 'BuildCore' as the reason for
>>> the trigger.
>>> It may seem that it is a violation of the blacklist in BuildApp, but a
>>> change in core/* is not the direct reason for BuildApp being triggered. It
>>> is the expected behaviour.
>>>
>>
>> I know it is the expected behaviour, but this behaviour could be improved
>> to better suit build requirements.
>>
>> If I were to move the "core" and "app" paths to different repos, with
>> "core" repo at revision "c01" and "app" repo at revision "a01", then I have
>> a nice workflow in Go, the way it was designed:
>> - If I commit "#c02" to the "core" repo, BuildCore will trigger. The new
>> "#c02" build of BuildCore will trigger BuildApp with the old "#a01"
>> revision from "app".
>> - If I commit "#a02" to the "app" repo, the AppBuild pipeline will build
>> "#a02" with "#c02". Subsequent commits to the app repo will build "#a03",
>> "#a04", etc with "#c02"
>>
>> This workflow is not possible when using paths in 1 repo for different
>> pipelines. I'm suggesting this is a shortfall in the workflow design -
>> something that could be improved to give the same flow in a monolithic repo
>> structure. The fan-in algorithm should not only trace materials, but also
>> factor in black / whitelists as part of the algorithm.
>>
>> I am able to work around this and get the desired workflow by using a
>> different hostname for the same repo in the 2 pipelines.
>>
>>
>>
>>>
>>> In this scenario, if a commit is pushed to "app/foo", both BuildCore and
>>>> BuildApp detect the new "repo" material. BuildCore doesn't trigger because
>>>> only "core" is in its whitelist ("app/foo" is blacklisted). BuildApp finds
>>>> the change, but doesn't trigger because it sees that BuildCore also uses
>>>> the "repo" material and wants to wait for it to trigger and finish - it
>>>> doesn't realize that BuildCore will not trigger due to the blacklist -
>>>> BuildApp never triggers automatically. I'd imagine that it's only when I
>>>> push to "core/bar" that BuildCore would trigger and that would then kick
>>>> off BuildApp...
>>>>
>>>
>>> Since BuildApp is dependent on BuildCore and git repository 'repo', the
>>> fan-in algorithm kicks in where it tries to find a compatible revision with
>>> both the git material 'repo' and BuildCore. Since it cannot find that
>>> BuildCore has been run with "app/foo", BuildApp will never get triggered.
>>> This behaviour exists irrespective of the blacklist and whitelist feature.
>>> And, because of the fan-in algorithm, the only valid change for BuildApp,
>>> as mentioned above to be triggered is any change to 'core/*' only.
>>>
>>>
>>>>
>>>> I now either need to remove the black / white lists and build BuildCore
>>>> for all commits or trigger manually. I'd imagine a solution to this is
>>>> that
>>>> BuildApp should not only check if "repo" is an upstream of "BuildCore" but
>>>> should also ensure that the new "repo" material is not blacklisted
>>>> there....
>>>>
>>>
>>> Yes, that is correct.
>>> The whitelist on the upstream pipeline "BuildCore" is respected even in
>>> BuildApp because of the dependency. A workaround would be to remove the
>>> whitelist in "BuildCore". However, keep in mind once the whitelist is
>>> removed, the blacklist in downstream "BuildApp" will not be respected (that
>>> is, blacklist on "BuildApp" is useless in this scenario to have) as any
>>> change triggering "BuildCore" will end up triggering "BuildApp" anyway.
>>>
>>> Thank You.
>>> Varsha
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
