If you want to make a job run concurrently, you call `concurrentBuild()` in the Jenkins groovy file.
I concur with Andrew that some jobs don't run concurrently due to GCP quota issues. This is the definitions without concurrentBuild() job_00_seed.groovy job_beam_Inventory.groovy job_beam_PerformanceTests_Dataflow.groovy job_beam_PerformanceTests_FileBasedIO_IT.groovy job_beam_PerformanceTests_FileBasedIO_IT_HDFS.groovy job_beam_PerformanceTests_HadoopInputFormat.groovy job_beam_PerformanceTests_JDBC.groovy job_beam_PerformanceTests_MongoDBIO_IT.groovy job_beam_PerformanceTests_Python.groovy job_beam_PerformanceTests_Spark.groovy job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy job_beam_PostCommit_Python_ValidatesContainer_Dataflow.groovy job_beam_PostCommit_Python_ValidatesRunner_Dataflow.groovy job_beam_PostCommit_Python_Verify.groovy job_beam_PreCommit_Website_Merge.groovy job_beam_PreCommit_Website_Stage.groovy job_seed_standalone.groovy It seems like we should be able to make all the non Dataflow ValidatesRunner concurrent builds. I'm not sure about the performance/website stuff. The seed jobs shouldn't be run in parallel. On Wed, May 2, 2018 at 1:33 PM Andrew Pilloud <[email protected]> wrote: > These jobs also require Dataflow which has various quotas on resource > usage. I hit these while working on the Dataflow Nexmark tests for SQL. I'm > not sure what the quota is on the account that Jenkins uses, but the > default quota will max out at around 2 concurrent jobs. > > Andrew > > On Wed, May 2, 2018 at 1:16 PM Scott Wegner <[email protected]> wrote: > >> While working on tuning our Gradle build for Jenkins, I noticed that our >> Jenkins jobs often get queued up, even though we have low utilization of >> our Jenkins executor pool [2]. For example, right now I see 4 instances of >> beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle queued [1], while the >> 'beam' label currently has 29 out of 32 executors available [2]. >> >> Is it possible to configure our Jenkins jobs to execute concurrently on >> separate nodes? I'm not sure whether it's safe for the same job to execute >> concurrently on the same node, for example if runner resources are shared. >> But I believe executing on separate nodes should be safe. >> >> Who would know more about this or have access to the Jenkins config if we >> did want to make changes? >> >> [1] >> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ >> [2] >> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ >> >> -- >> >> >> Got feedback? http://go/swegner-feedback >> >
