Hey,
I am trying to understand why you have set up the pipelines the way you
have. You have an upstream pipeline "BuildCore" and another pipeline
"BuildApp" dependent on 'BuildCore' and a git material 'repo' is common to
both. If your intention is to trigger the BuildCore for certain changes and
BuildApp for the rest of the changes in 'repo', can you please explain why
'BuildApp' should be dependent on 'BuildCore'?
Thank you.
On Monday, 25 July 2016 13:17:53 UTC+5:30, Brett Cave wrote:
>
>
>
> On Monday, 25 July 2016 08:50:07 UTC+2, Varsha Varadarajan wrote:
>>
>> The scenario:
>>>
>>>
>>> - A git repository "repo" with "core" and "app" root paths.
>>> - A "BuildCore" pipeline that uses "repo" with a whitelist on
>>> "core/*" - triggers automatically
>>> - A "BuildApp" pipeline that uses "repo" with a blacklist on
>>> "core/*" and an upstream pipeline dependency / material on "BuildCore" -
>>> triggers automatically
>>>
>>> The bug is that it seems like Go traces its upstream pipelines'
>>> materials and if a common material is found, the downstream pipeline will
>>> wait for the upstream pipeline to finish before triggering.
>>>
>>>
>>>
>> Given that the pipeline BuildApp is dependent on the git repository
>> 'repo' and pipeline 'BuildCore', any *valid* change in either materials
>> will trigger 'BuildApp'. In this scenario, a valid change is any change to
>> 'core/*'.
>> If you observe the build cause for BuildApp, when a commit is made in
>> 'core/*', the build cause will only list the 'BuildCore' as the reason for
>> the trigger.
>> It may seem that it is a violation of the blacklist in BuildApp, but a
>> change in core/* is not the direct reason for BuildApp being triggered. It
>> is the expected behaviour.
>>
>
> I know it is the expected behaviour, but this behaviour could be improved
> to better suit build requirements.
>
> If I were to move the "core" and "app" paths to different repos, with
> "core" repo at revision "c01" and "app" repo at revision "a01", then I have
> a nice workflow in Go, the way it was designed:
> - If I commit "#c02" to the "core" repo, BuildCore will trigger. The new
> "#c02" build of BuildCore will trigger BuildApp with the old "#a01"
> revision from "app".
> - If I commit "#a02" to the "app" repo, the AppBuild pipeline will build
> "#a02" with "#c02". Subsequent commits to the app repo will build "#a03",
> "#a04", etc with "#c02"
>
> This workflow is not possible when using paths in 1 repo for different
> pipelines. I'm suggesting this is a shortfall in the workflow design -
> something that could be improved to give the same flow in a monolithic repo
> structure. The fan-in algorithm should not only trace materials, but also
> factor in black / whitelists as part of the algorithm.
>
> I am able to work around this and get the desired workflow by using a
> different hostname for the same repo in the 2 pipelines.
>
>
>
>>
>> In this scenario, if a commit is pushed to "app/foo", both BuildCore and
>>> BuildApp detect the new "repo" material. BuildCore doesn't trigger because
>>> only "core" is in its whitelist ("app/foo" is blacklisted). BuildApp finds
>>> the change, but doesn't trigger because it sees that BuildCore also uses
>>> the "repo" material and wants to wait for it to trigger and finish - it
>>> doesn't realize that BuildCore will not trigger due to the blacklist -
>>> BuildApp never triggers automatically. I'd imagine that it's only when I
>>> push to "core/bar" that BuildCore would trigger and that would then kick
>>> off BuildApp...
>>>
>>
>> Since BuildApp is dependent on BuildCore and git repository 'repo', the
>> fan-in algorithm kicks in where it tries to find a compatible revision with
>> both the git material 'repo' and BuildCore. Since it cannot find that
>> BuildCore has been run with "app/foo", BuildApp will never get triggered.
>> This behaviour exists irrespective of the blacklist and whitelist feature.
>> And, because of the fan-in algorithm, the only valid change for BuildApp,
>> as mentioned above to be triggered is any change to 'core/*' only.
>>
>>
>>>
>>> I now either need to remove the black / white lists and build BuildCore
>>> for all commits or trigger manually. I'd imagine a solution to this is that
>>> BuildApp should not only check if "repo" is an upstream of "BuildCore" but
>>> should also ensure that the new "repo" material is not blacklisted there....
>>>
>>
>> Yes, that is correct.
>> The whitelist on the upstream pipeline "BuildCore" is respected even in
>> BuildApp because of the dependency. A workaround would be to remove the
>> whitelist in "BuildCore". However, keep in mind once the whitelist is
>> removed, the blacklist in downstream "BuildApp" will not be respected (that
>> is, blacklist on "BuildApp" is useless in this scenario to have) as any
>> change triggering "BuildCore" will end up triggering "BuildApp" anyway.
>>
>> Thank You.
>> Varsha
>>
>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
