Jason, your knowledge here is off. Parameters can be used in scripts, see a previous email I this thread that shows how it works.
On Mon, Jul 24, 2023, 4:11 PM Jason Smyth <[email protected]> wrote: > Hi Josh, > > I think there may be some confusion here regarding GoCD terminology and > common concepts. > > > i think the main source of confusion is that I thought parameters could > only be referred to in scripts! > > I didn't know you could refer to them inside of other configuration > properties! > > To the best of my knowledge, Parameters (GoCD concept) cannot be > referenced in scripts. You can call a script that uses parameters > (scripting concept), but as far as I know, GoCD Parameters are not > persisted in the Agent's runtime environment unless they are somehow passed > in via the Task definition. Are you sure you aren't thinking of Environment > Variables (GoCD concept)? Environment Variables can be defined in a few > different places in GoCD. As the name suggests, these values are persisted > in the Agent's runtime environment when a Task is executed. > > > I still have a question about how this works in examples using > templates. > > If we didn't define the pipeline parameter by default, how would gocd > interpret what I'm guessing would be a blank resource? > > If a Template references a Parameter then every Pipeline that uses that > Template _must_ define that Parameter. Depending on how the Parameter is > used in the Template, leaving the Parameter Value blank may be valid. > > In the case of using Parameters to define Resources, my testing shows that > each Parameter must define a single, valid, Resource. That is, if you want > to specify multiple Parameterized Resources, you must use multiple > Parameters. You cannot, for example, provide a Parameter Value of "foo, > bar" to make your Pipeline's Job depend on the "foo" and "bar" Resources. > GoCD rejects the configuration as invalid if you try to save it. Similarly, > GoCD rejects the configuration as invalid if a Parameter is used in the > Resource field and you try to leave its Value blank. > > Regarding your specific use case, you can solve it using either > Environments or Resources. The right solution depends on your requirements > and how you want to reason about your environment. > > The way I understand it, in the context of this discussion you have 2 > groups of Agents (Agents1 and Agents2) and 2 groups of Pipelines > (PipelinesA and PipelinesB). The Pipelines in PipelinesA can run on any > Agent, but the Pipelines in PipelinesB must run on the Agents in Agents2. > We will ignore the fact that Pipelines can contain multiple Stages and > multiple Jobs and assume either that all of the Pipelines contain a single > Stage with a single Job, or that the scheduling requirements are the same > for all Jobs in a given Pipeline. You have also talked about Pipeline > priority. > > Based on this, I assume your requirements are one of the following: > > 1. Agents should be used to the full extent possible; the workload in > PipelinesB is heavier so those Pipelines must not run on Agents1, or > 2. Pipelines in PipelinesB have a higher priority than those in > PipelinesA; Agents in Agents2 should take Jobs from PipelinesA only if > there are no pending Jobs for PipelinesB. > > GoCD supports the first scenario. You can achieve this by assigning 2 > Resources/Environments. Pipelines in PipelinesA get 1 Resource/Environment; > Pipelines in PipelinesB get the other. Agents in Agents1 get the PipelinesA > Resource/Environment; Agents in Agents2 get both. > > GoCD does not support the concept of priority, so scenario 2 is not > supported. The best you could accomplish would be to map each group of > Pipelines to a single group of Agents. > > Hope this helps. If I'm way off base it might help me better understand > your situation if you would provide snippets of your actual configuration. > > Cheers, > Jason > > > On Monday, 24 July 2023 at 11:43:58 UTC-4 Chad Wilson wrote: > >> On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <[email protected]> >> wrote: >> >>> >>> chad thanks for your answer. >>> >>> i think the main source of confusion is that I thought parameters could >>> only be referred to in scripts! >>> I didn't know you could refer to them inside of other configuration >>> properties! >>> Is this documented? Regardless that's super useful, there's probably >>> some other things that can be cleaned up knowing that. >>> >> >> >> https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters >> >> >> >>> I tried this on a pipeline w/out any template and it worked as >>> described. Just put the parameter reference in resource- UI accepts as >>> long as parameter exists and works. >>> >>> I still have a question about how this works in examples using templates. >>> If we didn't define the pipeline parameter by default, how would gocd >>> interpret what I'm guessing would be a blank resource? >>> >>> eg we have >>> >>> 1. a pipeline template called FAST_OR_SLOW_PIPE >>> 2. every pipeline implementing this template defines a parameter >>> called PIPE_RESOURCE_PARAM >>> >>> What happens if somebody only defines PIPE_RESOURCE_PARAM when the >>> pipeline is FAST? >>> If it's left as empty for ANY-aka-SLOW resources, will gocd >>> intepret this as a blank resource requirement and fail? >>> Or will it ignore blank resources? >>> >> >> I'm not sure - perhaps just try it empirically? It could either fail or >> see it as blank i.e "no resource requirement" - I don't think there's a >> strong case for either behaviour being more correct. >> >> -Chad >> >> >>> On Sun, Jul 23, 2023 at 10:04 AM Chad Wilson <[email protected]> >>> wrote: >>> >>>> With that description, if you want to use *environments >>>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#environment>* >>>> rather than resources >>>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>>> (and assuming you don't use environments for any other purpose), I would >>>> >>>> *1) Create 2 environments "fast" and "any"* >>>> >>>> *2) Map agents to environments* >>>> agents on GROUPA = machines that have less beefy hardware >>>> *- declare environments "any" when registered* >>>> >>>> agents on GROUPB = more expensive machines >>>> *- declare both environments "fast" and "any" when registered* >>>> >>>> *3) When configuring your pipelines* >>>> >>>> 1. have a couple of pipelines only run on the more expensive >>>> machines *<-- add these pipelines to "fast" environment* >>>> 2. have all other pipelines run in either group (next available >>>> agent) *<-- add these pipelines to "any" environment* >>>> >>>> This should give you roughly the semantics you say you want, but note >>>> it won't *prioritise* the GROUPB agents for use by the "couple of >>>> pipelines only run on the more expensive machines", it will just ensure >>>> they never run on the slower machines/agents. Something equivalent could >>>> also be done with resources >>>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>>> . >>>> >>>> There is no way to "try another agent" from inside the actual job's >>>> tasks. In this sense, the contents of tasks/scripts aren't relevant to >>>> scheduling. The GoCD resources and environments have to be known at >>>> schedule time. When you use pipeline parameters, they are realised at >>>> configuration time as when you create a pipeline from a template, it will >>>> force you to set the parameter values. >>>> >>>> To clarify, when you talked earlier about "a resource requirement" are >>>> you *actually* referring to GoCD's concept of resources, or were you >>>> talking in a generic sense? The answers are assuming you are talking about >>>> GoCD resources >>>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>>> but now I am more confused by your shell script. *If you want to use >>>> resources* (rather than environments) to affect scheduling, while >>>> still avoiding duplication of your templates, we are suggesting you use a >>>> parameter like *this*, not put it into some task content. You are >>>> setting the parameterized value into the field that determines the job's >>>> scheduling, not something that happens at execution time like a task. But >>>> again, if your goal is to control scheduling at pipeline level, for all >>>> jobs in a pipeline, you don't need to use resources, and can just use >>>> environments as in my earlier example above. >>>> >>>> [image: image.png] >>>> >>>> -Chad >>>> >>>> >>>> On Sun, Jul 23, 2023 at 8:21 PM Joshua Franta <[email protected]> >>>> wrote: >>>> >>>>> appreciate so many responses. i think we're a little apart so i'll >>>>> take the suggestion to give our example: >>>>> >>>>> GROUPA = machines that have less beefy hardware >>>>> GROUPB = more expensive machines >>>>> >>>>> we'd like to: >>>>> >>>>> 1. have a couple of pipelines only run on the more expensive >>>>> machines >>>>> 2. have all other pipelines run in either group (next available >>>>> agent) >>>>> >>>>> perhaps this was not clear from my previous explanations, but a couple >>>>> of people have suggested pipeline parameters. >>>>> >>>>> EXAMPLE >>>>> >>>>> a pipeline parameter is only going to be available to the job after >>>>> the job has already been assigned an agent, right? >>>>> >>>>> so if i have a pipeline called 'Priority' w/a parameter called >>>>> "group-id" and the pipeline has a 'Job' that is a shell script: >>>>> >>>>> ---- >>>>> ##!/bin/sh >>>>> >>>>> agent_resource="$GO_AGENT_RESOURCE_VARIABLE" >>>>> >>>>> if ! echo "#{group-id}" |grep -q "$agentr_esource"; then >>>>> >>>>> echo "agent can't run #{group-id} pipelines" >>>>> >>>>> ## won't this will make my pipeline fail when I want it to simply >>>>> try another agent? >>>>> exit 1 >>>>> fi >>>>> >>>>> ---- >>>>> >>>>> or perhaps people saying this know of some environment variable that >>>>> where we can request another agent? >>>>> >>>>> obviously pipeline parameters themseles don't do anything, so i'm >>>>> confused how i can affect assignment in a job that requires an agent >>>>> before >>>>> it runs. >>>>> this 2nd part is what i don't get above >>>>> >>>>> appreciate any clarifications or suggestions thx >>>>> >>>>> >>>>> On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> re: using environments rather than resources... environments can't >>>>>>> be defined at the pipeline level either though? >>>>>>> >>>>>> >>>>>> A pipeline is *assigned* to 0 or 1 environments (via the Admin > >>>>>> Environments UI if not using pipelines-as-code) - thus it's at the >>>>>> pipeline >>>>>> level by definition. It defines a scheduling requirement for all jobs in >>>>>> that pipeline. Which seems what you asked for with "able to communicate a >>>>>> resource requirement at the pipeline level" right? >>>>>> >>>>>> >>>>>> or i guess it's more correct to say that using environments is a bit >>>>>>> of a side-car feature, in that we use interact w/environments through a >>>>>>> different prisim/ui/config (no biggie) but also seems it's mutually >>>>>>> exclusive to maximizing overall usage of agents. for us if a given >>>>>>> host >>>>>>> can execute something (a pipeline, a job) it should. and if it can't, >>>>>>> it >>>>>>> shouldn't. >>>>>>> trying to force a hard divider can be useful for prod/qa staging, >>>>>>> but it seems to limit just being able to have pipelines declare their >>>>>>> needs. >>>>>>> maybe i'm missing what you're saying but i don't think environments >>>>>>> are functionally equivalent to resources? >>>>>>> >>>>>> >>>>>> I didn't imply they were functionally equivalent, but I did try to >>>>>> imply they were a different mechanism of defining a requirement on a >>>>>> job's >>>>>> scheduling, at the pipeline level. If a pipeline is assigned to an >>>>>> "environment", its jobs must be scheduled on agents that also declare >>>>>> they >>>>>> support that "environment". Similarly if a pipeline job declares a >>>>>> resource >>>>>> requirement, the agent must also have that resource declared for it to be >>>>>> assigned. This is a very similar, but different level of configuration >>>>>> of a >>>>>> scheduling requirement, no? >>>>>> https://docs.gocd.org/current/configuration/managing_environments.html >>>>>> >>>>>> Anyway, perhaps I don't understand what you are trying to achieve. If >>>>>> you are currently trying to "prioritise" pipelines by using resources you >>>>>> can also "prioritise" pipelines by having pools of agents, say, dedicated >>>>>> to an environment you call "high-priority". As I said, "Don't need to get >>>>>> hung up on the name [environment]". >>>>>> >>>>>> >>>>>> >>>>>>> we use template parameters extensively already. >>>>>>> eg we even templatize further inside our own jobs by re-using >>>>>>> scripts that interact with template parameters on most commonly used >>>>>>> templates (eg our most popular template has maybe 10-15 pipelines). >>>>>>> however this is more of a job specific thing since it's at the job >>>>>>> level. >>>>>>> if you're saying we could change every pipeline to read this at a >>>>>>> pipeline level is a non trivial change to every job. >>>>>>> >>>>>> >>>>>> You said you had many templates that varied only by the "resources" >>>>>> field for jobs. If that is the stated problem then parameters are a >>>>>> possible solution to remove duplication, no? >>>>>> >>>>>> >>>>>>> that's ok but i guess my overall question tho would be that if a >>>>>>> given job decided it couldn't execute the pipeline parameters... it has >>>>>>> no >>>>>>> way to pass the job to another agent? >>>>>>> >>>>>> >>>>>> That's the same problem you have currently if the resource is typoed >>>>>> or wrong inside the template, no? If the resource requirement has no >>>>>> available agents, then it can't be scheduled. >>>>>> >>>>>> >>>>>>> in such an example it would just fail the job, no? again maybe i'm >>>>>>> not following but this seems to not allow the business/value level to >>>>>>> declare minimum needs >>>>>>> (environments seem like they are more about maximimal requirements, >>>>>>> but i'm no expert) >>>>>>> >>>>>> >>>>>> I'm not following what you're trying to say here, sorry. >>>>>> >>>>>> Perhaps this would be easier if you gave a specific example of how >>>>>> you achieve "have some pipelines that are given higher preferences for >>>>>> agent/build resources" currently, rather than talking in abstract terms? >>>>>> >>>>>> -Chad >>>>>> >>>>>> >>>>>> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Have you tried to use "environments" (or a mix of environments and >>>>>>>> resources) to achieve what you are trying to? >>>>>>>> >>>>>>>> When scheduling jobs it's the combination of the resource and the >>>>>>>> environment that are matched to an agent, but the relevant environment >>>>>>>> is >>>>>>>> declared at the pipeline level like you refer to. Don't need to get >>>>>>>> hung up >>>>>>>> on the name so much. Yes, you can have "environment variables" >>>>>>>> attached to >>>>>>>> an environment and propagate those to all pipelines within it, but you >>>>>>>> don't have to use them like that. >>>>>>>> >>>>>>>> Alternatively, to make the templates less duplicated and allow the >>>>>>>> resource to flow from the pipeline *using* the template, you could >>>>>>>> try using template parameters >>>>>>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >>>>>>>> in the resources field? e.g #{job-resoure-requirement}? If there are >>>>>>>> only a >>>>>>>> small number of different resources used across the stages/jobs, you >>>>>>>> could >>>>>>>> use the parameters to "model" this I imagine. >>>>>>>> >>>>>>>> -Chad >>>>>>>> >>>>>>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <[email protected]> wrote: >>>>>>>> >>>>>>>>> QUESTION: >>>>>>>>> >>>>>>>>> Shouldn't we also be able to communicate a resource requirement at >>>>>>>>> the pipeline level, and not just inside a single job? >>>>>>>>> >>>>>>>>> I get that it definately needs to be at the job level since that's >>>>>>>>> the smallest unit of work and some machines can't execute certain >>>>>>>>> tasks. >>>>>>>>> But at the value-stream/pipeline/business level, you also want to >>>>>>>>> be able to have some pipelines compiling on preferred resources, no? >>>>>>>>> >>>>>>>>> >>>>>>>>> is there a better way to accomplish this? >>>>>>>>> or perhaps this already is possible and i'm missing it. >>>>>>>>> i looked closely at the config since sometimes you can do >>>>>>>>> something simple that is not possible inside the UI, but I'm not >>>>>>>>> seeing it. >>>>>>>>> >>>>>>>>> To restate use case: We have some pipelines that are given higher >>>>>>>>> preferences for agent/build resources. Wanting to do a lot more of >>>>>>>>> this, >>>>>>>>> but it's tricky because resources can only be defined at the job >>>>>>>>> level (in >>>>>>>>> the UI). Also we use a lot of templates, so having resources at >>>>>>>>> job >>>>>>>>> level means we end up having lots of alsomost identical templates >>>>>>>>> that only >>>>>>>>> vary by the resources used (which somewhat defeats the point of the >>>>>>>>> templates and the value of gocd in this respect). >>>>>>>>> >>>>>>>>> hoping there is a config hack or maybe i'm missinig something. >>>>>>>>> also if this could be done in a plugin, any color there would be >>>>>>>>> helpful (and i would make sure it's open sourced if need be). >>>>>>>>> >>>>>>>>> thx >>>>>>>>> >>>>>>>>> ps i keep using other ci/cd products and gocd is still one of the >>>>>>>>> all around bests. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "go-cd" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "go-cd" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "go-cd" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "go-cd" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "go-cd" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/43cb7edd-5e0d-4b06-9dac-024e40509d43n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/43cb7edd-5e0d-4b06-9dac-024e40509d43n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtp%3DquwcOTJv4jh8xKXf6JpOv5vHADqNkAwqaCS2yEZ%2BtA%40mail.gmail.com.
