chad thanks for your answer. i think the main source of confusion is that I thought parameters could only be referred to in scripts! I didn't know you could refer to them inside of other configuration properties! Is this documented? Regardless that's super useful, there's probably some other things that can be cleaned up knowing that.
I tried this on a pipeline w/out any template and it worked as described. Just put the parameter reference in resource- UI accepts as long as parameter exists and works. I still have a question about how this works in examples using templates. If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource? eg we have 1. a pipeline template called FAST_OR_SLOW_PIPE 2. every pipeline implementing this template defines a parameter called PIPE_RESOURCE_PARAM What happens if somebody only defines PIPE_RESOURCE_PARAM when the pipeline is FAST? If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this as a blank resource requirement and fail? Or will it ignore blank resources? It's not a big deal to define it anywhere, but I'm curious how it works when it's undefined since we didn't have this use case previously. If anybody knows would appreciate sharing. For the time being I think this is a better solution than environments, which seems a bit more opinionated and perhaps limiting for other solutions. However chad's demonstrate of the a non-traditional use of environments is helpful for thinking about other ways to use them. thanks chad and everybody. On Sun, Jul 23, 2023 at 10:04 AM Chad Wilson <[email protected]> wrote: > With that description, if you want to use *environments > <https://docs.gocd.org/current/introduction/concepts_in_go.html#environment>* > rather than resources > <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> > (and assuming you don't use environments for any other purpose), I would > > *1) Create 2 environments "fast" and "any"* > > *2) Map agents to environments* > agents on GROUPA = machines that have less beefy hardware > *- declare environments "any" when registered* > > agents on GROUPB = more expensive machines > *- declare both environments "fast" and "any" when registered* > > *3) When configuring your pipelines* > > 1. have a couple of pipelines only run on the more expensive machines *<-- > add these pipelines to "fast" environment* > 2. have all other pipelines run in either group (next available agent) *<-- > add these pipelines to "any" environment* > > This should give you roughly the semantics you say you want, but note it > won't *prioritise* the GROUPB agents for use by the "couple of pipelines > only run on the more expensive machines", it will just ensure they never > run on the slower machines/agents. Something equivalent could also be done > with resources > <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> > . > > There is no way to "try another agent" from inside the actual job's tasks. > In this sense, the contents of tasks/scripts aren't relevant to scheduling. > The GoCD resources and environments have to be known at schedule time. When > you use pipeline parameters, they are realised at configuration time as > when you create a pipeline from a template, it will force you to set the > parameter values. > > To clarify, when you talked earlier about "a resource requirement" are you > *actually* referring to GoCD's concept of resources, or were you talking > in a generic sense? The answers are assuming you are talking about GoCD > resources > <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> > but now I am more confused by your shell script. *If you want to use > resources* (rather than environments) to affect scheduling, while still > avoiding duplication of your templates, we are suggesting you use a > parameter like *this*, not put it into some task content. You are setting > the parameterized value into the field that determines the job's > scheduling, not something that happens at execution time like a task. But > again, if your goal is to control scheduling at pipeline level, for all > jobs in a pipeline, you don't need to use resources, and can just use > environments as in my earlier example above. > > [image: image.png] > > -Chad > > > On Sun, Jul 23, 2023 at 8:21 PM Joshua Franta <[email protected]> wrote: > >> appreciate so many responses. i think we're a little apart so i'll take >> the suggestion to give our example: >> >> GROUPA = machines that have less beefy hardware >> GROUPB = more expensive machines >> >> we'd like to: >> >> 1. have a couple of pipelines only run on the more expensive machines >> 2. have all other pipelines run in either group (next available agent) >> >> perhaps this was not clear from my previous explanations, but a couple of >> people have suggested pipeline parameters. >> >> EXAMPLE >> >> a pipeline parameter is only going to be available to the job after the >> job has already been assigned an agent, right? >> >> so if i have a pipeline called 'Priority' w/a parameter called >> "group-id" and the pipeline has a 'Job' that is a shell script: >> >> ---- >> ##!/bin/sh >> >> agent_resource="$GO_AGENT_RESOURCE_VARIABLE" >> >> if ! echo "#{group-id}" |grep -q "$agentr_esource"; then >> >> echo "agent can't run #{group-id} pipelines" >> >> ## won't this will make my pipeline fail when I want it to simply try >> another agent? >> exit 1 >> fi >> >> ---- >> >> or perhaps people saying this know of some environment variable that >> where we can request another agent? >> >> obviously pipeline parameters themseles don't do anything, so i'm >> confused how i can affect assignment in a job that requires an agent before >> it runs. >> this 2nd part is what i don't get above >> >> appreciate any clarifications or suggestions thx >> >> >> On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <[email protected]> >> wrote: >> >>> >>> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <[email protected]> >>> wrote: >>> >>>> re: using environments rather than resources... environments can't be >>>> defined at the pipeline level either though? >>>> >>> >>> A pipeline is *assigned* to 0 or 1 environments (via the Admin > >>> Environments UI if not using pipelines-as-code) - thus it's at the pipeline >>> level by definition. It defines a scheduling requirement for all jobs in >>> that pipeline. Which seems what you asked for with "able to communicate a >>> resource requirement at the pipeline level" right? >>> >>> >>> or i guess it's more correct to say that using environments is a bit of >>>> a side-car feature, in that we use interact w/environments through a >>>> different prisim/ui/config (no biggie) but also seems it's mutually >>>> exclusive to maximizing overall usage of agents. for us if a given host >>>> can execute something (a pipeline, a job) it should. and if it can't, it >>>> shouldn't. >>>> trying to force a hard divider can be useful for prod/qa staging, but >>>> it seems to limit just being able to have pipelines declare their needs. >>>> maybe i'm missing what you're saying but i don't think environments are >>>> functionally equivalent to resources? >>>> >>> >>> I didn't imply they were functionally equivalent, but I did try to imply >>> they were a different mechanism of defining a requirement on a job's >>> scheduling, at the pipeline level. If a pipeline is assigned to an >>> "environment", its jobs must be scheduled on agents that also declare they >>> support that "environment". Similarly if a pipeline job declares a resource >>> requirement, the agent must also have that resource declared for it to be >>> assigned. This is a very similar, but different level of configuration of a >>> scheduling requirement, no? >>> https://docs.gocd.org/current/configuration/managing_environments.html >>> >>> Anyway, perhaps I don't understand what you are trying to achieve. If >>> you are currently trying to "prioritise" pipelines by using resources you >>> can also "prioritise" pipelines by having pools of agents, say, dedicated >>> to an environment you call "high-priority". As I said, "Don't need to get >>> hung up on the name [environment]". >>> >>> >>> >>>> we use template parameters extensively already. >>>> eg we even templatize further inside our own jobs by re-using scripts >>>> that interact with template parameters on most commonly used templates (eg >>>> our most popular template has maybe 10-15 pipelines). >>>> however this is more of a job specific thing since it's at the job >>>> level. >>>> if you're saying we could change every pipeline to read this at a >>>> pipeline level is a non trivial change to every job. >>>> >>> >>> You said you had many templates that varied only by the "resources" >>> field for jobs. If that is the stated problem then parameters are a >>> possible solution to remove duplication, no? >>> >>> >>>> that's ok but i guess my overall question tho would be that if a given >>>> job decided it couldn't execute the pipeline parameters... it has no way to >>>> pass the job to another agent? >>>> >>> >>> That's the same problem you have currently if the resource is typoed or >>> wrong inside the template, no? If the resource requirement has no available >>> agents, then it can't be scheduled. >>> >>> >>>> in such an example it would just fail the job, no? again maybe i'm >>>> not following but this seems to not allow the business/value level to >>>> declare minimum needs >>>> (environments seem like they are more about maximimal requirements, but >>>> i'm no expert) >>>> >>> >>> I'm not following what you're trying to say here, sorry. >>> >>> Perhaps this would be easier if you gave a specific example of how you >>> achieve "have some pipelines that are given higher preferences for >>> agent/build resources" currently, rather than talking in abstract terms? >>> >>> -Chad >>> >>> >>> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <[email protected]> >>>> wrote: >>>> >>>>> Have you tried to use "environments" (or a mix of environments and >>>>> resources) to achieve what you are trying to? >>>>> >>>>> When scheduling jobs it's the combination of the resource and the >>>>> environment that are matched to an agent, but the relevant environment is >>>>> declared at the pipeline level like you refer to. Don't need to get hung >>>>> up >>>>> on the name so much. Yes, you can have "environment variables" attached to >>>>> an environment and propagate those to all pipelines within it, but you >>>>> don't have to use them like that. >>>>> >>>>> Alternatively, to make the templates less duplicated and allow the >>>>> resource to flow from the pipeline *using* the template, you could >>>>> try using template parameters >>>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >>>>> in the resources field? e.g #{job-resoure-requirement}? If there are only >>>>> a >>>>> small number of different resources used across the stages/jobs, you could >>>>> use the parameters to "model" this I imagine. >>>>> >>>>> -Chad >>>>> >>>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <[email protected]> wrote: >>>>> >>>>>> QUESTION: >>>>>> >>>>>> Shouldn't we also be able to communicate a resource requirement at >>>>>> the pipeline level, and not just inside a single job? >>>>>> >>>>>> I get that it definately needs to be at the job level since that's >>>>>> the smallest unit of work and some machines can't execute certain tasks. >>>>>> But at the value-stream/pipeline/business level, you also want to be >>>>>> able to have some pipelines compiling on preferred resources, no? >>>>>> >>>>>> >>>>>> is there a better way to accomplish this? >>>>>> or perhaps this already is possible and i'm missing it. >>>>>> i looked closely at the config since sometimes you can do something >>>>>> simple that is not possible inside the UI, but I'm not seeing it. >>>>>> >>>>>> To restate use case: We have some pipelines that are given higher >>>>>> preferences for agent/build resources. Wanting to do a lot more of >>>>>> this, >>>>>> but it's tricky because resources can only be defined at the job level >>>>>> (in >>>>>> the UI). Also we use a lot of templates, so having resources at job >>>>>> level means we end up having lots of alsomost identical templates that >>>>>> only >>>>>> vary by the resources used (which somewhat defeats the point of the >>>>>> templates and the value of gocd in this respect). >>>>>> >>>>>> hoping there is a config hack or maybe i'm missinig something. >>>>>> also if this could be done in a plugin, any color there would be >>>>>> helpful (and i would make sure it's open sourced if need be). >>>>>> >>>>>> thx >>>>>> >>>>>> ps i keep using other ci/cd products and gocd is still one of the all >>>>>> around bests. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "go-cd" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "go-cd" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com.
