Consider this: a. Various pipelines publish their artifacts as packages (e.g. RPMs). b. You run integration tests using the "latest" of the various pipelines. c. When a given set of artifacts work together (as per the integration tests), you publish this information somewhere. d. You run a "deploy" task on all the agents for that environment. e. The deploy task just runs a configuration management tool like Ansible, which uses the version numbers provided, and then "ensures" that those versions are present.
This way, if a given computer will always have the version that you want it to have. -- Ram On Thu, Oct 6, 2016 at 2:18 AM, 'Anant Agarwal' via go-cd < [email protected]> wrote: > Thanks Jyoti so there are two key points here: > 1. publish the changed installers to a package repository > 2. DeploymentService1' pipeline could now be changed to depend on this > package repo instead with appropriate package spec (say Service1) specified. > > Your idea looks good however I am not sure about point2 "appropriate > package spec" I will explore this configuration and will get back. > > Thanks a lot. > > > Sent from my iPhone > > On 5 Oct 2016, at 12:32, Jyoti Singh <[email protected]> wrote: > > Just thinking out loud, what if the integration test pipeline was > refactored to have two stages. First stage to run the integration-tests and > second to publish the changed installers to a package repository ( > https://www.go.cd/plugins/). > 'DeploymentService1' pipeline could now be changed to depend on this > package repo instead with appropriate package spec (say Service1) > specified. The same goes for the other two deployment pipelines - > DeploymentService2 and DeploymentService3. > With this, the deployment pipelines would get triggered only if a new > package matching the specified specs was published. Won't that help simply > the scenario? > > On Wednesday, 5 October 2016 15:59:54 UTC+5:30, Anant Agarwal wrote: >> >> May be https://api.go.cd/current/#scheduling-pipelines is the answer >> for controlling the deployment pipelines trigger. >> >> >> On Wednesday, October 5, 2016 at 9:50:05 AM UTC+1, Anant Agarwal wrote: >>> >>> Hi Aravind >>> >>> Thank you very for your response. I like the idea you have suggested >>> but unable to comment on the code snippet you have attached. >>> Could you clarify following in bold "as an artefact to all three >>> downstream pipelines, *which can figure out whether to re-deploy or >>> not.*" does the pipeline has ability to figure out to run or not >>> based on artefact or are you suggesting that deployment code can decide >>> whether to deploy or not in which case all deployment pipeline will run but >>> deployment code will deploy conditionally. >>> >>> I was thinking that may be I have to rethink how I setup the pipeline, >>> I am finding it hard to believe that such a common scenario in today's >>> word about building microservices, running integration test and deployment >>> of what has changed should be easy. >>> >>> Thanks >>> Anant Agarwal >>> >>> >>> On Monday, October 3, 2016 at 9:51:08 PM UTC+1, Aravind SV wrote: >>>> >>>> No, there's no way, as far as I know. I remember that this question (or >>>> a very similar one) has come up before. I think what's making this hard to >>>> do is the fact that the information shown in the build cause popup of the >>>> "IntegrationTest" pipeline is lost. I mean, if you click on "Changes" in >>>> that pipeline, it'll show you, in yellow, which upstream pipeline(s) caused >>>> it ("IntegrationTest" pipeline) to trigger. But, once that pipeline has >>>> finished, that information is no longer accessible to the downstream >>>> pipelines (DeploymentService1, ...). >>>> >>>> I'm wondering: If GoCD provided an environment variable (say, >>>> "GO_material_name_CHANGED" like "GO_material_name_LABEL" etc), then a >>>> deployment descriptor (a file describing what happened) can be created and >>>> pushed down as an artifact to all three downstream pipelines, which can >>>> figure out whether to re-deploy or not. >>>> >>>> I know it's not a pipeline modeling solution but it is a solution which >>>> can enable this kind of a workflow to be supported. Thoughts? >>>> >>>> >>>> I'm thinking of a code change like this (just to clarify): >>>> >>>> diff --git >>>> a/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java >>>> b/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java >>>> index 2513d68..3847bf2 100644--- >>>> a/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java+++ >>>> b/common/src/com/thoughtworks/go/config/materials/PackageMaterial.java@@ >>>> -168,6 +168,7 @@ public class PackageMaterial extends AbstractMaterial { >>>> @Override >>>> public void populateEnvironmentContext(EnvironmentVariableContext >>>> context, MaterialRevision materialRevision, File workingDir) { >>>> context.setProperty(upperCase(format("GO_PACKAGE_%s_LABEL", >>>> escapeEnvironmentVariable(getName().toString()))), >>>> materialRevision.getRevision().getRevision(), false);+ >>>> context.setProperty(upperCase(format("GO_PACKAGE_%s_CHANGED", >>>> escapeEnvironmentVariable(getName().toString()))), >>>> Boolean.toString(materialRevision.isChanged()), false); >>>> for (ConfigurationProperty configurationProperty : >>>> getPackageDefinition().getRepository().getConfiguration()) { >>>> >>>> context.setProperty(getEnvironmentVariableKey("GO_REPO_%s_%s", >>>> configurationProperty.getConfigurationKey().getName()), >>>> configurationProperty.getValue(), >>>> configurationProperty.isSecure()); >>>> diff --git >>>> a/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java >>>> >>>> b/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java >>>> index 391bfae..1190cf9 100644--- >>>> a/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java+++ >>>> >>>> b/common/src/com/thoughtworks/go/config/materials/PluggableSCMMaterial.java@@ >>>> -265,6 +265,7 @@ public class PluggableSCMMaterial extends >>>> AbstractMaterial { >>>> @Override >>>> public void populateEnvironmentContext(EnvironmentVariableContext >>>> context, MaterialRevision materialRevision, File workingDir) { >>>> context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", >>>> "LABEL"), materialRevision.getRevision().getRevision(), false);+ >>>> context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", "CHANGED"), >>>> Boolean.toString(materialRevision.isChanged()), false); >>>> for (ConfigurationProperty configurationProperty : >>>> scmConfig.getConfiguration()) { >>>> context.setProperty(getEnvironmentVariableKey("GO_SCM_%s_%s", >>>> configurationProperty.getConfigurationKey().getName()), >>>> configurationProperty.getValue(), >>>> configurationProperty.isSecure()); >>>> diff --git >>>> a/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java >>>> b/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java >>>> index d3c86db..aa9509e 100644--- >>>> a/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java+++ >>>> b/common/src/com/thoughtworks/go/config/materials/ScmMaterial.java@@ >>>> -129,13 +129,14 @@ public abstract class ScmMaterial extends >>>> AbstractMaterial { >>>> String toRevision = materialRevision.getRevision().getRevision(); >>>> String fromRevision = >>>> materialRevision.getOldestRevision().getRevision(); >>>> - setGoRevisionVariables(environmentVariableContext, fromRevision, >>>> toRevision);+ setGoRevisionVariables(environmentVariableContext, >>>> materialRevision, fromRevision, toRevision); >>>> } >>>> - private void setGoRevisionVariables(EnvironmentVariableContext >>>> environmentVariableContext, String fromRevision, String toRevision) {+ >>>> private void setGoRevisionVariables(EnvironmentVariableContext >>>> environmentVariableContext, MaterialRevision materialRevision, String >>>> fromRevision, String toRevision) { >>>> setVariableWithName(environmentVariableContext, toRevision, >>>> GO_REVISION); >>>> setVariableWithName(environmentVariableContext, toRevision, >>>> GO_TO_REVISION); >>>> setVariableWithName(environmentVariableContext, fromRevision, >>>> GO_FROM_REVISION);+ setVariableWithName(environmentVariableContext, >>>> Boolean.toString(materialRevision.isChanged()), "GO_CHANGED"); >>>> } >>>> >>>> protected void setVariableWithName(EnvironmentVariableContext >>>> environmentVariableContext, String value, String propertyName) { >>>> diff --git >>>> a/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java >>>> >>>> b/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java >>>> index 016e049..ea3681a 100644--- >>>> a/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java+++ >>>> >>>> b/common/src/com/thoughtworks/go/config/materials/dependency/DependencyMaterial.java@@ >>>> -160,6 +160,7 @@ public class DependencyMaterial extends AbstractMaterial >>>> { >>>> DependencyMaterialRevision revision = >>>> (DependencyMaterialRevision) materialRevision.getRevision(); >>>> context.setPropertyWithEscape(format("GO_DEPENDENCY_LABEL_%s", >>>> getName()), revision.getPipelineLabel()); >>>> context.setPropertyWithEscape(format("GO_DEPENDENCY_LOCATOR_%s", >>>> getName()), revision.getRevision());+ >>>> context.setPropertyWithEscape(format("GO_DEPENDENCY_CHANGED_%s", >>>> getName()), Boolean.toString(materialRevision.isChanged())); >>>> } >>>> >>>> public boolean isAutoUpdate() { >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Oct 3, 2016 at 6:01 AM, 'Anant Agarwal' via go-cd < >>>> [email protected]> wrote: >>>> >>>>> Hi >>>>> >>>>> We have a bunch of micro services in separate code repositories. The >>>>> pipelines we have setup look like the diagram below. >>>>> >>>>> <https://lh3.googleusercontent.com/-n18NACBsIBY/V_IrFsQqmCI/AAAAAAAAAmU/IwkT90HzjAcdeJXPz22XnMJlHtxJXd6TQCLcB/s1600/Screen%2BShot%2B2016-10-03%2Bat%2B10.51.46.png> >>>>> >>>>> At the moment whenever we make changes to any one service let's say >>>>> service1, integration test pipeline runs and then triggers deployment of >>>>> all services. >>>>> >>>>> What we want is that Integration test pipeline should run if changes >>>>> to any of the dependent services but it should trigger deployment only for >>>>> the particular service which originated the change. Is that possible? >>>>> >>>>> so we are kind of looking to achieve "AND" condition in pipeline >>>>> trigger. i.e. Deployment service 1 pipeline should trigger only when >>>>> Service1 Artefact was built AND integration test passed. and this should >>>>> not trigger Deployment service2 pipeline. >>>>> >>>>> >>>>> Appreciate your help. >>>>> >>>>> >>>>> Thanks >>>>> Anant Agarwal >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "go-cd" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/go-cd/z1t0fy1Ldck/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
