Using Aravind's method I've managed to improve the build pipeline:
# Our job invokes a wrapper batch file that splits the list of inputs and 
extracts the correct parameters (eg: CONFIGURATION_LIST="x64 x64 ARM ARM", 
PLATFORM_LIST="Debug Release Debug Release", run_instances=4). It then 
invokes the actual build script
# Upload any build artifacts from all runInstances to the runInstance-1 job 
using the RESTful API. This means that the output artifacts we expect from 
each stage are defined within the code we're building - this is probably a 
positive change
# Downstream jobs can fetch artifacts from the runInstances-1 job using 
whatever filter is convenient

PROS:
* Expected artifacts are checked into our repository as part of the build 
script. Even if this changes over time we can still build earlier versions
* Duplication of configuration and build logic is minimized
* Adding or removing configurations is straightforward and only requires 
changes to the single build job

CONS:
* Artifacts must be uploaded as individual files - there doesn't seem to be 
a way to upload an entire folder at once
* Changes to the configuration matrix break the ability to build older code 
revisions

While I was eventually able to get the repo-config working I found the 
process of making changes to be too error prone to make sense. There's no 
mechanism to validate the configuration files or preview what jobs will be 
created with a given input, and no visual editor. It's unfortunate, because 
being able to check in the job configuration itself would take care of the 
second point which I expect will be the biggest source of problems down the 
road.

Thanks again for the suggestion!

On Wednesday, February 21, 2018 at 12:46:37 PM UTC-6, Andrew Jones wrote:
>
> This seems like an elegant way to handle it. It does create more of a 
> dependency between the submitted scripts and configuration than I'd like - 
> if I add or remove a configuration I need to increase/decrease the number 
> of agents. Attempting to rebuild older code would either fail or miss some 
> needed configurations. We work around this by allocating more agents than 
> we think we'll ever need and handling out of bounds JOB_RUN_INDEX values in 
> the script.
> Definitely a good approach to consider - thanks for the suggestion.
>
> On Wednesday, February 21, 2018 at 12:39:23 PM UTC-6, Aravind SV wrote:
>>
>> I would keep it simple and do a run X instances 
>> <https://docs.gocd.org/current/advanced_usage/admin_spawn_multiple_jobs.html#run-x-instances-of-a-job>
>>  
>> and use the GO_JOB_RUN_INDEX and GO_JOB_RUN_COUNT to decide what to do. 
>> Even something as simple as:
>>
>> $ cat matrix.rb
>> #!env ruby
>>
>> matrix = {
>>   "1" => "Release/x64",
>>   "2" => "Release/RISC",
>>   "3" => "Debug/x64",
>>   "4" => "Debug/RISC",
>> }
>>
>> puts "Execute: make #{matrix[ENV['GO_JOB_RUN_INDEX']]}"
>>
>> $ GO_JOB_RUN_INDEX=1 ruby matrix.rb
>> Execute: make Release/x64
>>
>> $ GO_JOB_RUN_INDEX=3 ruby matrix.rb
>> Execute: make Debug/x64
>>
>> If you now just setup a job with "Run 4 instances" and have it call: ruby 
>> matrix.rb, you'll get 4 GoCD jobs, running parallel-ly, depending on agents 
>> available, since GoCD sets the correct GO_JOB_RUN_INDEX. No dependency on 
>> plugins. You can even run this locally.
>>
>>
>>
>> On Wed, Feb 21, 2018 at 12:33 PM, Ketan Padegaonkar <
>> ketanpad...@gmail.com> wrote:
>>
>>> I've been working on a groovy based DSL to define your pipeline. Since 
>>> it's all code, you can create the usual programming constructs to generate 
>>> a matrix.
>>>
>>> The code and some binaries are available, please read this note 
>>> <https://github.com/ketan/gocd-groovy-dsl-config-plugin#a-note-about-security>
>>>  
>>> before you use it.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to