Short term I am working on limiting concurrent execution of jobs in a flow to
prevent exhaustion (there is no point pulling myjob#2 from the queue when
myjob#3 is in the queue as well - think parameterised jobs and build("myjob",
upstream:build; other:fred) ).
Not sure to understand, can you describe with more details ?
Sure, hopefully I can explain this better with an example.
Take an example flow that is triggered by a "commit" job.
build("level1_smoke", run: upstream)
build("level2_tests", run: upstream)
build("level3_indepthtests", run: upstream)
assuming
a "commit" build take 10 minutes
level1_smoke build takes 5 minutes
level2_tests build take 20 minutes
level3_indepth build takes 1 hour
(you want to manage the flow with the flow plugin - why wouldn't you)
Given any commit you want to know quickly if the level1_smoke tests fail (or
pass).
As such you need to enable concurrent executions of flow_job
As commits can be hitting the SCM for "commit" constantly we will have a run of
"commit" every 10 minutes, without any restrictions (given that the jobs are
parameterised) we will very quickly have lots of entries for the subsequent
jobs.
As we want some quick turnaround (we want to know as soon as possible when a
commit doesn't pass all these tests) we need to enable concurrent exectution of
these level_* jobs.
This will very quicky exhaust a large number of executors.
In reality - you don't care about knowing if every build of "commit" passes all
the tests.
What you care about is knowing that when level3_indepth tests have run against
a particular commit build that all the preceding jobs in the flow have also
passed and run against the exact same commit run and that any invocation of the
level3_indepth tests will run against the most recent commit that has also
passed the preceding level2_tests.
The non-flow way could be for level3_indepth to get the latest run of "commit"
as it started and to be triggered by a successful level2_test. This has a
drawback as you could get a run for which there is no successful level2 tests.
Alternatively you can use promoted builds - but that punts logic up to the
commit build which may be undesirable for various reasons.
Or you could get the run from the leve2_test that caused the level3_indepth to
run - this way makes it harder to use some Jenkins plugins, and moves a lot of
the intelligence to the Job (which should just be concentrating on the test
task at hand).
So back to the BuildFlow - We have multiple FlowRuns running concurrently and
say level3_indepth is running we don't want to start another one. However as
soon as level3_indepth has finished we want the most recent attempt to start a
new level3_indepth to succeed (and any that are not the most recent to abort
the flow run)
Doing this in the Flow DSL makes sense as there you can take advantage of
things extra flow features like parallel / guard & rescue etc, showing the
build graph etc...
So the aboce leads to something like the following:
build("level1_smoke", run: upstream)
block ("level2") {
build("level2_tests", run: upstream)
}
block ("level3") {
build("level3_indepthtests", run: upstream)
}
Now this makes even more sense when you have realise that all the tests are
cucumber tests just with different tags - and these can actually be the same
job - (thus reducing job maintainance) and can be parameterised further...
Which leads to something like the following..
build("acceptance_tests", run: upstream, level:"1")
block ("level2") {
build("acceptance_tests", run: upstream, level:"2")
}
block ("level3") {
parallel {
build("acceptance_tests", run: upstream, level:"3a")
build("acceptance_tests", run: upstream, level:"3b")
}
}
This will run acceptance_tests with level:"1" for every invocation of the
BuildFlow. However it will only run acceptance_tests for the latest level:"2"
if a level:"2" job is not already running.
If it is not running the the flowrun will wait. If multiple FlowRuns are
waiting for the "level2" block then when the block is released only the most
recent FlowRun will be able to enter it - and all others will have the FlowRun
Aborted. The block can also run across multiple jobs for example the
"level3" block will prevent any new starts of 3a or 3b tests.
Or in a time line (at is the acceptance_tests job)
t=0 flow#1 -> at level=1
t=5 flow#1 -> at level=2
t=10 flow#2 -> at level=1
t=20 flow#3 -> at level=1
t=25 flow#1 -> [ at level=3a, at level=3b ]
t=25 flow#2 (aborted as flow#3 is waiting to enter the same block)
t=25 flow#3 -> at level=2
t=65 flow#3 -> [ at level=3a, at level=3b ]
Hopefully that clears things up a bit. If something is not clear let me know
(as I will eventually need to put something like this on the Wiki)
/James
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
