[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope Kevin Phillips whatever you are seeing is an unrelated issue. Please do not discuss it here. Rather, file a fresh issue with steps to reproduce from scratch in workflow-scm-step-plugin. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Kevin Phillips commented on JENKINS-38268 Re: Parallel step and closure scope So I guess the problem as described in my example above seems to be particular to the `git` and `checkout` build steps, however I have managed to reproduce the problem using the `sh` step as well - just in a more elaborate way. I haven't reduced that one to a minimal reproducible example yet so I left that part out of my reply. For now the only easy-to-reproduce example I have is with those 2 specific build steps. Also, the "trick" of renaming the variable names doesn't seem to work with either of those two build steps as I had originally thought. It does, however, seem to work around the problem wrt the `sh` build step that I have also encountered in some of my production builds. It's not a "fix" of course but it has temporarily worked around the problem for me in at least one use case. I have others I need to investigate further to see if / how they relate. Will post back as soon as I know more. Regardless, the example script I provided above should reproduce the problem in that one context at least. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jacob Keller commented on JENKINS-38268 Re: Parallel step and closure scope so you're using the git step, and expect its output to return something unique? Is it possible that after that runs once it always returns the first branch? I suspect maybe that step isn't actually returning the value you think it does? Hmm, but you said if you rename the variable it works fine?? that's odd. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Kevin Phillips edited a comment on JENKINS-38268
Re: Parallel step and closure scope
Hmmm - I'm not sure if my case is exactly the same as this one, but it appears to me that this bug - or one similar to it - still exists. I'm running Jenkins v2.148 and Pipeline plugin v2.6 and can still break this parallel variable expansion problem, without using closures. Here's an example: {code:java}parallel ("thread1": {node() {def a = git(branch: 'BranchA', credentialsId: 'MyCreds', url: 'url/to/git/repo.git')def b = "1"def c = sh(returnStdout: true, script: 'echo 1').trim() echo "git commit " + a.GIT_COMMITecho "b is " + becho "c is " + c}},"thread2": {node() {def a = git(branch: 'BranchB', credentialsId: 'MyCreds', url: 'url/to/git/repo.git')def b = "2"def c = sh(returnStdout: true, script: 'echo 2').trim()echo "git commit " + a.GIT_COMMITecho "b is " + becho "c is " + c}}){code}In this example, I would expect the output to be something like{noformat}[thread1] git commit [thread1] b is 1[thread1] c is 1[thread2] git commit [thread2] b is 2[thread2] c is 2{noformat}What I actually get is this:{noformat}[thread1] git commit [thread1] b is 1[thread1] c is 1[thread2] git commit [thread2] b is 2[thread2] c is 2{noformat}While the 'b' and 'c' variables seem to get unique values for each of the parallel threads, the 'a' value becomes ambiguous, always returning a single Git commit hash.Further, I've confirmed that which Git commit hash that is shown represents the one that is executed "first" in the parallel stages. So, for example, if you put a small "sleep 5" at the top of thread1, then you get the Git commit has from BranchB in both echo statements, and if you move the sleep statement to the top of thread2 you get the has from BranchA.Also interesting is that although the 'a' variable seems to get confused based on the thread execution, the other variables do not. I thought at first that maybe assigning a static value to the variable may have explained the discrepancy like in the case of the 'b' variable so I added a dynamically executed shell step to the mix for variable 'c' and I get the same results as b... so I'm not sure how / why / what is going on here. Oh, and the other interesting thing I found as well is that if I rename the variables in the example code snippet so they are all unique names (ie: a,b,c in thread 1 and x,y,z in thread 2) the behavior results in the expected output as I mentioned above. So not only does this problem appear to be isolated to parallel threads, and to dynamic build steps, but also the variable names as well.
Add Comment
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Kevin Phillips commented on JENKINS-38268
Re: Parallel step and closure scope
Hmmm - I'm not sure if my case is exactly the same as this one, but it appears to me that this bug - or one similar to it - still exists. I'm running Jenkins v2.148 and Pipeline plugin v2.6 and can still break this parallel variable expansion problem, without using closures. Here's an example:
parallel (
"thread1": {
node() {
def a = git(branch: 'BranchA', credentialsId: 'MyCreds', url: 'url/to/git/repo.git')
def b = "1"
def c = sh(returnStdout: true, script: 'echo 1').trim()
echo "git commit " + a.GIT_COMMIT
echo "b is " + b
echo "c is " + c
}
},
"thread2": {
node() {
def a = git(branch: 'BranchB', credentialsId: 'MyCreds', url: 'url/to/git/repo.git')
def b = "2"
def c = sh(returnStdout: true, script: 'echo 2').trim()
echo "git commit " + a.GIT_COMMIT
echo "b is " + b
echo "c is " + c
}
}
)
In this example, I would expect the output to be something like
[thread1] git commit
[thread1] b is 1
[thread1] c is 1
[thread2] git commit
[thread2] b is 2
[thread2] c is 2
What I actually get is this:
[thread1] git commit
[thread1] b is 1
[thread1] c is 1
[thread2] git commit
[thread2] b is 2
[thread2] c is 2
While the 'b' and 'c' variables seem to get unique values for each of the parallel threads, the 'a' value becomes ambiguous, always returning a single Git commit hash. Further, I've confirmed that which Git commit hash that is shown represents the one that is executed "first" in the parallel stages. So, for example, if you put a small "sleep 5" at the top of thread1, then you get the Git commit has from BranchB in both echo statements, and if you move the sleep statement to the top of thread2 you get the has from BranchA. Also interesting is that although the 'a' variable seems to get confused based on the thread execution, the other variables do not. I thought at first that maybe assigning a static value to the variable may have explained the discrepancy like in the case of the 'b' variable so I added a dynamically executed shell step to the mix for variable 'c' and I get the same results as b... so I'm not sure how / why / what is going on here. Oh, and the other interesting thing I found as well is that if I rename the variables in the example code snippet so they are all unique names (ie: a,b,c in thread 1 and x,y,z in thread 2) the behavior results in the expected output as I mentioned above. So not only does this problem appear to be isolated to parallel threads, and to dynamic build steps, but also the
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
John Elion edited a comment on JENKINS-38268
Re: Parallel step and closure scope
I've been hit by the same issue. I've noticed that certain kinds of scoping passes through ok, but some kinds do not. def say1(s) \{ println('say1: ' + s) } def say2 = \{ s -> println('say2: ' + s) } def setup = \{ def map = [:] def say3 = \{ s -> println('say3: ' + s) } for (def i = 0; i < 3; i++) \{ def x = I map[x] = \{ println x; say1( x ); say2( x ); say3( x ) } } return map } def jobs = setup() parallel(jobs)The "println" , and "say1" , and work but "say2" work but and "say3" does not (and the def say4( x ) \{ are wrong, println( x ) } form isn't allowed inside setup's block) and in a different ways . Stripping all the "noise", I get the following: [0] 0 [1] 1 [2] 2 [0] say1: 0 [1] say1: 1 [2] say1: 2 [0] say2: 2 [1] say2: 0 [2] say2: 1 [0] say3: 2 [1] say3: 2 [2] say3: 2 Hoping this helps say2 is not printing the x value associated with a workaround the parallel branch. say3 is always printing the x value from branch "2". (I don't have mine yet had originally commented that say2 was working, but it is not. The branch name in brackets should always match the number at the end of each line. ) Spotting these differences helped me work around the issue and also understanding what the scope resolution issue might be...
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
John Elion edited a comment on JENKINS-38268
Re: Parallel step and closure scope
I've been hit by the same issue. I've noticed that certain kinds of scoping passes through ok, but some kinds do not. def say1(s) \{ println('say1: ' + s) } def say2 = \{ s -> println('say2: ' + s) } def setup = \{ def map = [:] def say3 = \{ s -> println('say3: ' + s) } for (def i = 0; i < 3; i++) \{ def x = I map[x] = \{ println x; say1(x); say2(x); say3(x) } } return map } def jobs = setup() parallel(jobs)The "println", "say1", and "say2" work but "say3" does not (and the def say4(x) \{ println(x) } form isn't allowed inside setup's block). Stripping all the "noise", I get the following: [0] 0 [1] 1 [2] 2 [0] say1: 0 [1] say1: 1 [2] say1: 2 [0] say2: 2 [1] say2: 0 [2] say2: 1 [0] say3: 2 [1] say3: 2 [2] say3: 2Hoping this helps with a workaround (I don't have mine yet) and also understanding what the scope resolution issue might be...
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
John Elion commented on JENKINS-38268
Re: Parallel step and closure scope
I've been hit by the same issue. I've noticed that certain kinds of scoping passes through ok, but some kinds do not. def say1(s) { println('say1: ' + s) } def say2 = { s -> println('say2: ' + s) } def setup = { def map = [:] def say3 = { s -> println('say3: ' + s) } for (def i = 0; i < 3; i++) { def x = I map[x] = { println x; say1; say2; say3 } } return map } def jobs = setup() parallel(jobs) The "println", "say1", and "say2" work but "say3" does not (and the def say4 { println } form isn't allowed inside setup's block). Stripping all the "noise", I get the following: [0] 0 [1] 1 [2] 2 [0] say1: 0 [1] say1: 1 [2] say1: 2 [0] say2: 2 [1] say2: 0 [2] say2: 1 [0] say3: 2 [1] say3: 2 [2] say3: 2 Hoping this helps with a workaround (I don't have mine yet) and also understanding what the scope resolution issue might be...
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick updated JENKINS-38268 Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Jesse Glick Status: Reopened Open Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Jesse Glick Priority: Major Critical Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope Most likely a bug in the depths of groovy-cps. Probably easy enough to reproduce in a unit test there. After that, I have no idea offhand how difficult the fix would be. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Torben Knerr edited a comment on JENKINS-38268
Re: Parallel step and closure scope
I believe [~nvgoldin]'s example is not representative for this issue, since `get_dummy_params` is a method. If it were closure instead (which is totally not needed in this case, so it would be quite artificial) it would fail again, I suspect.Here is another example which suffers from the issue, and I have totally no clue how to work around it:{code:groovy}def onEachSlave(doStuff) { def doStuffClosures = [:] for (slave in ['slavelnx1', 'slavelnx2', 'slavelnx3']) {def s = slavedoStuffClosures[s] = { echo "running on ${s}..."; doStuff(s); echo "...done on ${s}!" } } return doStuffClosures}parallel(onEachSlave { slave ->echo " do doing stuff on ${slave}..."}){code}The output is:{noformat}[Pipeline] parallel[Pipeline] [slavelnx1] { (Branch: slavelnx1)[Pipeline] [slavelnx2] { (Branch: slavelnx2)[Pipeline] [slavelnx3] { (Branch: slavelnx3)[Pipeline] [slavelnx1] echo[slavelnx1] running on slavelnx1...[Pipeline] [slavelnx2] echo[slavelnx2] running on slavelnx2...[Pipeline] [slavelnx3] echo[slavelnx3] running on slavelnx3...[Pipeline] [slavelnx1] echo[slavelnx1] doing stuff on slavelnx3[Pipeline] [slavelnx1] echo[slavelnx1] ...done on slavelnx1![Pipeline] [slavelnx1] }[Pipeline] [slavelnx2] echo[slavelnx2] doing stuff on slavelnx3[Pipeline] [slavelnx2] echo[slavelnx2] ...done on slavelnx2![Pipeline] [slavelnx2] }[Pipeline] [slavelnx3] echo[slavelnx3] doing stuff on slavelnx3[Pipeline] [slavelnx3] echo[slavelnx3] ...done on slavelnx3![Pipeline] [slavelnx3] }[Pipeline] // parallel[Pipeline] End of Pipeline{noformat}This is Jenkins 2.32.1 and workflow-cps 2.23[~jglick] can you confirm this is the issue, or is it just my missing groovy foo?Would also be happy to hear if someone can come up with a workaround for this
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Torben Knerr edited a comment on JENKINS-38268
Re: Parallel step and closure scope
I believe [~nvgoldin]'s example is not representative for this issue, since `get_dummy_params` is a method. If it were closure instead (which is totally not needed in this case, so it would be quite artificial) it would fail again, I suspect.Here is another example which suffers from the issue, and I have totally no clue how to work around it:{code:groovy}def onEachSlave(doStuff) { def slaves doStuffClosures = [ :] for (slave in [ 'slavelnx1', 'slavelnx2', 'slavelnx3'] def doStuffClosures = [:] for (slave in slaves ) {def s = slavedoStuffClosures[s] = { echo "running on ${s}..."; doStuff(s); echo "...done on ${s}!" } } return doStuffClosures}parallel(onEachSlave { slave ->echo "do stuff on ${slave}..."}){code}The output is:{noformat}[Pipeline] parallel[Pipeline] [slavelnx1] { (Branch: slavelnx1)[Pipeline] [slavelnx2] { (Branch: slavelnx2)[Pipeline] [slavelnx3] { (Branch: slavelnx3)[Pipeline] [slavelnx1] echo[slavelnx1] running on slavelnx1...[Pipeline] [slavelnx2] echo[slavelnx2] running on slavelnx2...[Pipeline] [slavelnx3] echo[slavelnx3] running on slavelnx3...[Pipeline] [slavelnx1] echo[slavelnx1] doing stuff on slavelnx3[Pipeline] [slavelnx1] echo[slavelnx1] ...done on slavelnx1![Pipeline] [slavelnx1] }[Pipeline] [slavelnx2] echo[slavelnx2] doing stuff on slavelnx3[Pipeline] [slavelnx2] echo[slavelnx2] ...done on slavelnx2![Pipeline] [slavelnx2] }[Pipeline] [slavelnx3] echo[slavelnx3] doing stuff on slavelnx3[Pipeline] [slavelnx3] echo[slavelnx3] ...done on slavelnx3![Pipeline] [slavelnx3] }[Pipeline] // parallel[Pipeline] End of Pipeline{noformat}This is Jenkins 2.32.1 and workflow-cps 2.23[~jglick] can you confirm this is the issue, or is it just my missing groovy foo?Would also be happy to hear if someone can come up with a workaround for this
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Torben Knerr edited a comment on JENKINS-38268
Re: Parallel step and closure scope
I believe [~nvgoldin]'s example is not representative for this issue, since `get_dummy_params` is a method. If it were closure instead (which is totally not needed in this case, so it would be quite artificial) it would fail again, I suspect. [~jglick] here Here is another example which suffers from the issue, and I have totally no clue how to work around it:{code:groovy}def onEachSlave(doStuff) { def slaves = ['slavelnx1', 'slavelnx2', 'slavelnx3'] def doStuffClosures = [:] for (slave in slaves) {def s = slavedoStuffClosures[s] = { echo "running on ${s}..."; doStuff(s); echo "...done on ${s}!" } } return doStuffClosures}parallel(onEachSlave { slave ->echo "do stuff on ${slave}..."}){code}The output is:{noformat}[Pipeline] parallel[Pipeline] [slavelnx1] { (Branch: slavelnx1)[Pipeline] [slavelnx2] { (Branch: slavelnx2)[Pipeline] [slavelnx3] { (Branch: slavelnx3)[Pipeline] [slavelnx1] echo[slavelnx1] running on slavelnx1...[Pipeline] [slavelnx2] echo[slavelnx2] running on slavelnx2...[Pipeline] [slavelnx3] echo[slavelnx3] running on slavelnx3...[Pipeline] [slavelnx1] echo[slavelnx1] doing stuff on slavelnx3[Pipeline] [slavelnx1] echo[slavelnx1] ...done on slavelnx1![Pipeline] [slavelnx1] }[Pipeline] [slavelnx2] echo[slavelnx2] doing stuff on slavelnx3[Pipeline] [slavelnx2] echo[slavelnx2] ...done on slavelnx2![Pipeline] [slavelnx2] }[Pipeline] [slavelnx3] echo[slavelnx3] doing stuff on slavelnx3[Pipeline] [slavelnx3] echo[slavelnx3] ...done on slavelnx3![Pipeline] [slavelnx3] }[Pipeline] // parallel[Pipeline] End of Pipeline{noformat}This is Jenkins 2.32.1 and workflow-cps 2.23 [~jglick] can you confirm this is the issue, or is it just my missing groovy foo?Would also be happy to hear if someone can come up with a workaround for this
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Torben Knerr commented on JENKINS-38268
Re: Parallel step and closure scope
I believe Nadav Goldin's example is not representative for this issue, since `get_dummy_params` is a method. If it were closure instead (which is totally not needed in this case, so it would be quite artificial) it would fail again, I suspect. Jesse Glick here is another example which suffers from the issue, and I have totally no clue how to work around it:
Unable to find source-code formatter for language: groovy. Available languages are: actionscript, html, java, _javascript_, none, sql, xhtml, xml
def onEachSlave(doStuff) {
def slaves = ['slavelnx1', 'slavelnx2', 'slavelnx3']
def doStuffClosures = [:]
for (slave in slaves) {
def s = slave
doStuffClosures[s] = { echo "running on ${s}..."; doStuff(s); echo "...done on ${s}!" }
}
return doStuffClosures
}
parallel(onEachSlave { slave ->
echo "do stuff on ${slave}..."
})
The output is:
[Pipeline] parallel
[Pipeline] [slavelnx1] { (Branch: slavelnx1)
[Pipeline] [slavelnx2] { (Branch: slavelnx2)
[Pipeline] [slavelnx3] { (Branch: slavelnx3)
[Pipeline] [slavelnx1] echo
[slavelnx1] running on slavelnx1...
[Pipeline] [slavelnx2] echo
[slavelnx2] running on slavelnx2...
[Pipeline] [slavelnx3] echo
[slavelnx3] running on slavelnx3...
[Pipeline] [slavelnx1] echo
[slavelnx1] doing stuff on slavelnx3
[Pipeline] [slavelnx1] echo
[slavelnx1] ...done on slavelnx1!
[Pipeline] [slavelnx1] }
[Pipeline] [slavelnx2] echo
[slavelnx2] doing stuff on slavelnx3
[Pipeline] [slavelnx2] echo
[slavelnx2] ...done on slavelnx2!
[Pipeline] [slavelnx2] }
[Pipeline] [slavelnx3] echo
[slavelnx3] doing stuff on slavelnx3
[Pipeline] [slavelnx3] echo
[slavelnx3] ...done on slavelnx3!
[Pipeline] [slavelnx3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
This is Jenkins 2.32.1 and workflow-cps 2.23
Add Comment
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Jesse Glick commented on JENKINS-38268
Re: Parallel step and closure scope
the looping variable needs to be aliased, otherwise only the last variable will be sent to all closures
Yes this is just a general aspect of Groovy.
$ groovy -e 'def closures = []; for (int i = 0; i < 10; i++) {closures += { -> println i}}; closures.each {it()}'
10
10
10
10
10
10
10
10
10
10
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Nadav Goldin commented on JENKINS-38268
Re: Parallel step and closure scope
Jesse Glick, thanks! I changed the code to simple for loops without iterators , and without '@NonCPS' and it seems to work . A small gotcha I encountered is that the looping variable needs to be aliased, otherwise only the last variable will be sent to all closures(unlike when using iterators, I think). For reference, here is the code that does work:
def get_dummy_params(val)
{
return [string(name: 'dummy', value: "$val")]
}
def create_jobs()
{
def jobs = [:]
for (int i=1; i <= 3; i++) {
def x = i
jobs["job-$x"] = { -> build([job: "job-$x", parameters: get_dummy_params(x) ]) }
}
return jobs
}
stage ('triggering') {
parallel(create_jobs())
}
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope Nadav Goldin create_jobs is wrong. You may not call steps like build from inside a method marked @NonCPS. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Nadav Goldin commented on JENKINS-38268
Re: Parallel step and closure scope
Hi, I'm experiencing what I think is a symptom of the same issue, while trying to trigger parallel builds, take this pipeline job, assuming the dummy jobs 'job-1,2,3' exists:
@NonCPS
def get_dummy_params(val)
{
return [string(name: 'dummy', value: "$val")]
}
@NonCPS
def create_jobs()
{
def jobs = [:]
(1..3).each { jobs["job-$it"] = { -> build([job: "job-$it", parameters: get_dummy_params(it) ]) } }
return jobs
}
stage ('triggering') {
parallel(create_jobs())
}
I would expect it to trigger the 3 jobs in parallel, instead it 'nests' under 'job-1' branch all 3 jobs, and creates empty branches for 'job-2' and 'job-3', causing the job to never end. Console output:
Started by user admin
[Pipeline] stage
[Pipeline] { (triggering)
[Pipeline] parallel
[Pipeline] [job-1] { (Branch: job-1)
[Pipeline] [job-1] build (Building job-1)
[job-1] Scheduling project: job-1
[Pipeline] [job-2] { (Branch: job-2)
[Pipeline] [job-1] build (Building job-2)
[job-1] Scheduling project: job-2
[Pipeline] [job-3] { (Branch: job-3)
[Pipeline] [job-1] build (Building job-3)
[job-1] Scheduling project: job-3 !pipeline_steps.png|thumbnail!
[job-1] Starting building: job-3 #1
...
I found only 2 workarounds: 1. Don't use any functions to generate data fed into 'parallel' function(i.e. create everything under the 'stage' step). 2. Feeding a newly created map to parallel and redefining the groovy closures with a '.call()' (frankly I have no idea how I came up with this). I.e. replace the 'stage' in the previous example with:
stage ('triggering') {
def jobs = create_jobs()
parallel([
'job-1': { jobs['job-1'].call() },
'job-2': { jobs['job-2'].call() },
'job-3': { jobs['job-3'].call() },
])
}
Output as expected:
Started by user admin
[Pipeline] stage
[Pipeline] { (triggering)
[Pipeline] parallel
[Pipeline] [job-1] { (Branch: job-1)
[Pipeline] [job-2] { (Branch: job-2)
[Pipeline] [job-3] { (Branch: job-3)
[Pipeline] [job-1] build (Building job-1)
[job-1] Scheduling project: job-1
[Pipeline] [job-2] build (Building job-2)
[job-2] Scheduling project: job-2
[Pipeline] [job-3] build (Building job-3)
[job-3] Scheduling project: job-3
[job-1] Starting building: job-1 #2
[job-2] Starting building: job-2 #2
[Pipeline] [job-2] }
[job-3] Starting building: job-3 #2
[Pipeline] [job-1] }
[Pipeline] [job-3] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Nadav Goldin updated an issue Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Nadav Goldin Attachment: pipeline_steps.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
jlpinardon commented on JENKINS-38268
Re: Parallel step and closure scope
Same problem when trying to loop over a map of git repositories : def scmRepo = [ 'jobRepo': [ 'url': "." 'branch': "origin/master", 'credentialsId': '.', 'targetDir': 'src' ], 'envRepo': [ 'url': "." 'branch': "origin/master", 'credentialsId': '.', 'targetDir': 'src' ], 'srcRepo': [ 'url': "." 'branch': "origin/master", 'credentialsId': '.', 'targetDir': 'src' ] ] @NonCPS def runCheckOut(scmRepo) { def branches = [: ] scmRepo.each() { repoName, repoRef - > branches["$ {repoName} "] = { node { println "\nCheckout content from $ {repoRef.url} " //your logic there } } } parallel branches } When using runCheckOut, it runs but never ends. No error message is displayed.
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Joe George commented on JENKINS-38268 Re: Parallel step and closure scope We experience this too, it's quite annoying and hard to work around. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope I wonder if ClosureCallEnv is misbehaving. Need to go through this in a debugger. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Andras Kovi commented on JENKINS-38268
Re: Parallel step and closure scope
Seems like the names of the parameters of the called closures play some role. For example:
def finallyHandler(param, closure) {
echo "HANDLER:param=$param"
closure(param, param)
}
work = [
"1": { finallyHandler(1) { p, p1 -> echo "p=$p p1=$p1" } },
"2": { finallyHandler(2) { p, p2 -> echo "p=$p p2=$p2" } },
"3": { finallyHandler(3) { p, p3 -> echo "p=$p p3=$p3" } },
]
parallel work
Output:
[Pipeline] parallel
[Pipeline] [1] { (Branch: 1)
[Pipeline] [2] { (Branch: 2)
[Pipeline] [3] { (Branch: 3)
[Pipeline] [1] echo
"[1] HANDLER:param=1"
[Pipeline] [2] echo
"[2] HANDLER:param=2"
[Pipeline] [3] echo
"[3] HANDLER:param=3"
[Pipeline] [1] echo
"[1] p=3 p1=1"
[Pipeline] [1] }
[Pipeline] [2] echo
"[2] p=3 p2=2"
[Pipeline] [2] }
[Pipeline] [3] echo
"[3] p=3 p3=3"
[Pipeline] [3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Andras Kovi edited a comment on JENKINS-38268
Re: Parallel step and closure scope
Seems like the names of the parameters of the called closures play some role. For example:{code:java}def finallyHandler(param, closure) {echo "HANDLER:param=$param"closure(param, param)}work = [ "1": { finallyHandler(1) { p, p1 -> echo "p=$p p1=$p1" } }, "2": { finallyHandler(2) { p, p2 -> echo "p=$p p2=$p2" } }, "3": { finallyHandler(3) { p, p3 -> echo "p=$p p3=$p3" } },]parallel work{code}Output: (we want to see the same value for p= and pX=) {code:java}[Pipeline] parallel[Pipeline] [1] { (Branch: 1)[Pipeline] [2] { (Branch: 2)[Pipeline] [3] { (Branch: 3)[Pipeline] [1] echo"[1] HANDLER:param=1"[Pipeline] [2] echo"[2] HANDLER:param=2"[Pipeline] [3] echo"[3] HANDLER:param=3"[Pipeline] [1] echo"[1] p=3 p1=1"[Pipeline] [1] }[Pipeline] [2] echo"[2] p=3 p2=2"[Pipeline] [2] }[Pipeline] [3] echo"[3] p=3 p3=3"[Pipeline] [3] }[Pipeline] // parallel[Pipeline] End of PipelineFinished: SUCCESS{code}
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope No idea offhand, will have to reproduce in a functional test and study in a debugger. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Alexander Olofsson commented on JENKINS-38268 Re: Parallel step and closure scope To be honest, currying was just a good way to showcase the issue. Anything that embeds a value into the closure seems to be lost in parallelisation, so even wrapping the closure into another one that feeds it the value as a static parameter doesn't work. (file30 and file31 in the example) Ran into this when trying to create a list of parallel steps out of files from a folder, to spread out a series of fixed-time workloads onto many executors so that the build step wouldn't take as much time. Tried everything I could think of to work around the problem, not even rolling my own closure class to store the value worked. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope Well that is more likely an issue with curry; not sure if that was ever implemented. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Alexander Olofsson commented on JENKINS-38268
Re: Parallel step and closure scope
Been able to replicate this consistently without having to add a node block, in fact I still haven't found a way to not replicate it. Composited closures also seem to be even more unhappy about being parallelized. Script:
def heavyweight_task = { fn->
println "Doing heavy work on ${fn}..."
}
def work = [
file10: heavyweight_task.curry('file10'),
file11: heavyweight_task.curry('file11'),
file20: ({'file20'} >> heavyweight_task),
file21: ({'file21'} >> heavyweight_task),
file30: { heavyweight_task.call('file30') },
file31: { heavyweight_task.call('file31') },
file40: heavyweight_task.clone().curry('file40'),
file41: heavyweight_task.clone().curry('file41'),
file50: heavyweight_task.dehydrate().curry('file50'),
file51: heavyweight_task.dehydrate().curry('file51'),
]
parallel(work)
output:
[Pipeline] parallel
[Pipeline] [file10] { (Branch: file10)
[Pipeline] [file11] { (Branch: file11)
[Pipeline] [file20] { (Branch: file20)
[Pipeline] [file21] { (Branch: file21)
[Pipeline] [file30] { (Branch: file30)
[Pipeline] [file31] { (Branch: file31)
[Pipeline] [file40] { (Branch: file40)
[Pipeline] [file41] { (Branch: file41)
[Pipeline] [file50] { (Branch: file50)
[Pipeline] [file51] { (Branch: file51)
[Pipeline] [file10] echo
[file10] Doing heavy work on file51...
[Pipeline] [file10] }
[Pipeline] [file11] echo
[file11] Doing heavy work on file51...
[Pipeline] [file11] }
[Pipeline] [file20] }
[Pipeline] [file21] }
[Pipeline] [file40] echo
[file40] Doing heavy work on file31...
[Pipeline] [file40] }
[Pipeline] [file41] echo
[file41] Doing heavy work on file31...
[Pipeline] [file41] }
[Pipeline] [file50] echo
[file50] Doing heavy work on file31...
[Pipeline] [file50] }
[Pipeline] [file51] echo
[file51] Doing heavy work on file31...
[Pipeline] [file51] }
[Pipeline] [file30] echo
[file30] Doing heavy work on file31...
[Pipeline] [file30] }
[Pipeline] [file31] echo
[file31] Doing heavy work on file31...
[Pipeline] [file31] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Add Comment
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick reopened an issue I doubt this is a duplicate of JENKINS-26481. call in this case should refer to the implicit method defined in CPS-transformed sources, not a binary method. And it is the receiver, not an argument, which is a closure. Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Jesse Glick Resolution: Duplicate Status: Resolved Reopened Assignee: Jesse Glick Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Andrew Bayer resolved as Duplicate Yet another of the JENKINS-26481 symptoms... Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Andrew Bayer Status: Open Resolved Resolution: Duplicate Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title James Femia commented on JENKINS-38268 Re: Parallel step and closure scope Same issue here in 2.19.1 LTS / Pipeline 2.4 Thought cloning the closure might be a workaround but it had no effect. Now trying to work around it with more copy and paste of code that would have been in the closure, instead. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick commented on JENKINS-38268 Re: Parallel step and closure scope Possibly a duplicate etc. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Jesse Glick Labels: parallel pipeline Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-38268 Parallel step and closure scope Change By: Jesse Glick Component/s: pipeline Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jacob Keller edited a comment on JENKINS-38268 Re: Parallel step and closure scope I have this exact issue as well . I have a block that takes a closure and sets up the parallel branches and wants to run the closure once for each node, but the parameter to the closure always takes only the last value when inside the node. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title Jacob Keller commented on JENKINS-38268 Re: Parallel step and closure scope I have this exact issue. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38268) Parallel step and closure scope
Title: Message Title
Phil Grayson created an issue
Jenkins / JENKINS-38268
Parallel step and closure scope
Issue Type:
Bug
Assignee:
Jesse Glick
Attachments:
console_output.png, console_output_no_node_step.png, pipeline_step.png
Components:
pipeline, workflow-cps-plugin
Created:
2016/Sep/16 10:42 AM
Environment:
Jenkins: 2.7.4 (affecting 1.6 too) Pipeline: 2.3 (tested on fresh install with latest dependencies) Environment setup; 1. Run "docker run -p 8080:8080 -p 5:5 jenkins" 2. Browse to http://localhost:8080 3. Perform initial jenkins setup - install 'pipeline' plugin
Labels:
pipeline parallel
Priority:
Major
Reporter:
Phil Grayson
I'm experiencing some odd behaviour with the parallel step related to variable scoping. The following minimal pipeline script demonstrates my problem.
def closure = { val ->
node { }
println val
}
parallel([
a: { closure.call('a') },
b: { closure.call('b') }
])
Expected output
