[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title Abdulla Hawara updated an issue Jenkins / JENKINS-49710 Pipelines run under heavy load sometimes hang running Docker Change By: Abdulla Hawara Component/s: durable-task-plugin Component/s: pipeline Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title
Abdulla Hawara edited a comment on JENKINS-49710
Re: Pipelines run under heavy load sometimes hang running Docker
My *theory* about the issue:Jenkins has `*Text file busy*` error coming from `*durable*` which is used by `*pipeline nodes and processes*` plugin. This issue is easily reproducible by running any command e.g. `*sh "echo 'Hello'"*` in any jenkins job many times in parallel. This hang is caused when running a docker container using the * docker plugin * , but when we run `*docker*` without manually using ` * sh docker ...* ` it never hangs , but that `*Text file busy*` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever {color:#ff}To reproduce {color}{code:java}Text file busy{code}error, please follow these steps: 1. Run Jenkins locally `docker run -p 8080:8080 -p 5:5 jenkins/jenkins:lts` ver. 2.89.4 2. install just the recommended plugins on it 3. create new pipeline projects called `*hello*` and `*hello2*` 4. put this code in ‘hello’ :{code:java}node{sh "echo 'hello'"}{code}and this in `*hello2*`:{code:java}node{sh "echo 'hello'"sleep(2)}{code}5. Create a new pipeline project called `*runner*` and put this inside:{code:java}COUNTER = 0node{def jobs = [:]// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}}// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}}// run them all 20 times in parallel20.times {parallel jobs}}def triggerProject(jobName) {return {try{build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]} catch (ex){println ex}}}{code}6. Go to your jenkins configurations and change the executors to `*50*` 7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands 8. run runner again and you will see that some of `*hello*` and `*hello2*` has `*text file busy*` errorThe logs you get afterwards:{code:java}Running in Durability level: MAX_SURVIVABILITY[Pipeline] nodeRunning on Jenkins in /var/jenkins_home/workspace/hello@6[Pipeline] {[Pipeline] sh[hello@6] Running shell scriptsh: 1: /var/jenkins_home/workspace/hello@6@tmp/durable-a771d7dd/script.sh: Text file busy[Pipeline] }[Pipeline] // node[Pipeline] End of PipelineERROR: script returned exit code 2Finished: FAILURE{code}
Add Comment
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title
Abdulla Hawara edited a comment on JENKINS-49710
Re: Pipelines run under heavy load sometimes hang running Docker
My *theory* about the issue:Jenkins has `*Text file busy*` error coming from `*durable*` which is used by `*pipeline nodes and processes*` plugin. This issue is easily reproducible by running any command e.g. `*sh "echo 'Hello'"*` in any jenkins job many times in parallel. This hang is caused when running a docker container using the *docker plugin*, but when we run `*docker*` manually using ` *sh ' docker ... ' * ` it never hangs, but that `*Text file busy*` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever {color:#ff}To reproduce {color}{code:java}Text file busy{code}error, please follow these steps: 1. Run Jenkins locally `docker run -p 8080:8080 -p 5:5 jenkins/jenkins:lts` ver. 2.89.4 2. install just the recommended plugins on it 3. create new pipeline projects called `*hello*` and `*hello2*` 4. put this code in ‘hello’ :{code:java}node{sh "echo 'hello'"}{code}and this in `*hello2*`:{code:java}node{sh "echo 'hello'"sleep(2)}{code}5. Create a new pipeline project called `*runner*` and put this inside:{code:java}COUNTER = 0node{def jobs = [:]// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}}// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}}// run them all 20 times in parallel20.times {parallel jobs}}def triggerProject(jobName) {return {try{build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]} catch (ex){println ex}}}{code}6. Go to your jenkins configurations and change the executors to `*50*` 7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands 8. run runner again and you will see that some of `*hello*` and `*hello2*` has `*text file busy*` errorThe logs you get afterwards:{code:java}Running in Durability level: MAX_SURVIVABILITY[Pipeline] nodeRunning on Jenkins in /var/jenkins_home/workspace/hello@6[Pipeline] {[Pipeline] sh[hello@6] Running shell scriptsh: 1: /var/jenkins_home/workspace/hello@6@tmp/durable-a771d7dd/script.sh: Text file busy[Pipeline] }[Pipeline] // node[Pipeline] End of PipelineERROR: script returned exit code 2Finished: FAILURE{code}
Add Comment
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title Abdulla Hawara updated an issue Jenkins / JENKINS-49710 Pipelines run under heavy load sometimes hang running Docker Change By: Abdulla Hawara Labels: groo Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title Abdulla Hawara updated an issue Jenkins / JENKINS-49710 Pipelines run under heavy load sometimes hang running Docker Change By: Abdulla Hawara Labels: groo Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title
Abdulla Hawara edited a comment on JENKINS-49710
Re: Pipelines run under heavy load sometimes hang running Docker
My *theory* about the issue:Jenkins has `*Text file busy*` error coming from `*durable*` which is used by `*pipeline nodes and processes*` plugin. This issue is easily reproducible by running any command e.g. `*sh "echo 'Hello'"*` in any jenkins job many times in parallel. This hang is caused when running a docker container using the *docker plugin* (the container will stay alive) , but when we run `*docker*` manually using *sh 'docker ...'* it never hangs, but that `*Text file busy*` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever {color:#ff}To reproduce {color}{code:java}Text file busy{code}error, please follow these steps: 1. Run Jenkins locally `docker run -p 8080:8080 -p 5:5 jenkins/jenkins:lts` ver. 2.89.4 2. install just the recommended plugins on it 3. create new pipeline projects called `*hello*` and `*hello2*` 4. put this code in ‘hello’ :{code:java}node{sh "echo 'hello'"}{code}and this in `*hello2*`:{code:java}node{sh "echo 'hello'"sleep(2)}{code}5. Create a new pipeline project called `*runner*` and put this inside:{code:java}COUNTER = 0node{def jobs = [:]// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}}// add 24 instances of the same test to run them later in parallel24.times {jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}}// run them all 20 times in parallel20.times {parallel jobs}}def triggerProject(jobName) {return {try{build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]} catch (ex){println ex}}}{code}6. Go to your jenkins configurations and change the executors to `*50*` 7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands 8. run runner again and you will see that some of `*hello*` and `*hello2*` has `*text file busy*` errorThe logs you get afterwards:{code:java}Running in Durability level: MAX_SURVIVABILITY[Pipeline] nodeRunning on Jenkins in /var/jenkins_home/workspace/hello@6[Pipeline] {[Pipeline] sh[hello@6] Running shell scriptsh: 1: /var/jenkins_home/workspace/hello@6@tmp/durable-a771d7dd/script.sh: Text file busy[Pipeline] }[Pipeline] // node[Pipeline] End of PipelineERROR: script returned exit code 2Finished: FAILURE{code}
Add Comment
[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker
Title: Message Title
Abdulla Hawara commented on JENKINS-49710
Re: Pipelines run under heavy load sometimes hang running Docker
My theory about the issue: Jenkins has `Text file busy` error coming from `durable` which is used by `pipeline nodes and processes` plugin. This issue is easily reproducible by running any command e.g. `sh "echo 'Hello'"` in any jenkins job many times in parallel. This hang is caused when running a docker container using the docker plugin, but when we run `docker` without using `sh` it never hangs but that `Text file busy` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever To reproduce
Text file busy
error, please follow these steps: 1. Run Jenkins locally `docker run -p 8080:8080 -p 5:5 jenkins/jenkins:lts` ver. 2.89.4 2. install just the recommended plugins on it 3. create new pipeline projects called `hello` and `hello2` 4. put this code in ‘hello’ :
node{
sh "echo 'hello'"
}
and this in `hello2`:
node{
sh "echo 'hello'"
sleep(2)
}
5. Create a new pipeline project called `runner` and put this inside:
COUNTER = 0
node{
def jobs = [:]
// add 24 instances of the same test to run them later in parallel
24.times {
jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}
}
// add 24 instances of the same test to run them later in parallel
24.times {
jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}
}
// run them all 20 times in parallel
20.times {
parallel jobs
}
}
def triggerProject(jobName) {
return {
try{
build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]
} catch (ex){
println ex
}
}
}
6. Go to your jenkins configurations and change the executors to `50` 7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands 8. run runner again and you will see that some of `hello` and `hello2` has `text file busy` error The logs you get afterwards:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in
