[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Dennis Keitzel commented on JENKINS-48490 Re: Intermittently slow docker provisioning with no errors For reference: We hit a similar issue where the cloud-stats-plugin was the cause. See JENKINS-56863. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title pjdarton resolved as Fixed I believe that, through careful use of "read timeout" and "pull timeout" on docker clouds & templates, coupled with the "avoid using broken clouds/templates" feature (all introduced in 1.1.4) this issue should now be fixed. We have a fairly busy Jenkins server with dozens of static nodes, lots of docker clouds, hundreds of jobs, and many dozens of builds running at any one time and, since adding this functionality, it all seems stable now (other than JENKINS-53621 which is a separate issue). TL;DR: I think it's fixed; re-open if it's still there in 1.1.6. Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: pjdarton Status: Reopened Resolved Assignee: Nicolas De Loof pjdarton Resolution: Fixed Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title pjdarton commented on JENKINS-48490 Re: Intermittently slow docker provisioning with no errors We used to encounter these kinds of issues. We eventually traced it to cloud plugins doing blocking operations within the main "provisioning" thread and node-disposal operations (which happen while a Jenkins core object is locked) were also sometimes taking a long time. These combined to causing Jenkins' ability to create and destroy slaves was severely impeded. This only happened when the servers that the plugins were communicating with weren't responding swiftly, but as Docker is prone to total lockups (and vSphere can take 4 hours to fail a 40millisecond operation), when all is not well, Jenkins also becomes unwell. It took a fair bit of work, but I made enhancements to both the vSphere plugin and the Docker plugin to reduce the amount of remote-API-calls made during the "provisioning" and "termination" process threads, and to ensure that everything had a timeout (so nothing would lock up forever). The vSphere plugin (version 2.16 onwards) contains my changes, but you'd have to get the bleeding-edge build of the docker-plugin from here for my changes to that (as we've not done a release of that yet). Note: If you take the docker plugin, make sure that you set a non-zero connection timeout, read timeout and (in the templates) pull timeout. Also, for pure speed, remove the instance caps (if you don't specify an instance cap, the plugin no longer counts the instances). That should ensure that nothing can cause the core Jenkins cloud resources to stay "locked" for a long period of time. See also: JENKINS-49235 as that has the potential to cause problems for a busy Jenkins server. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Matthew Ludlum commented on JENKINS-48490 Re: Intermittently slow docker provisioning with no errors I've been trying to reproduce this issue on my local box using SSH executors with both 1.1.2 and 1.1.3 to no avail. We are still seeing it on other instances. In the short term, I've thrown together a quick and dirty script job to "unstick" the jobs. https://gist.github.com/MattLud/1f8a56fcce933f7e97c366de54c85ba9 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Jason Swager commented on JENKINS-48490 Re: Intermittently slow docker provisioning with no errors We are seeing similar symptoms after upgrading the Docker plugin. Our Jenkins masters started seeing this problem after upgrading to Docker Plugin v1.1.1 and even v1.1.2. One big difference, we're using SSH to connect rather than JNLP. The larger and more busy the Jenkins master, the faster this problem occurs. Our larger one we had to downgrade the Docker plugin to it's prior version, 0.16.2. The smaller Jenkins don't suffer immediately from the problem and a restart of them clears the problem - at least for another couple days. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title David van Laatum reopened an issue Same thing for me Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: David van Laatum Resolution: Fixed Status: Resolved Reopened Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov edited a comment on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally *able to reproduce this* in an isolated Jenkins installation running the following versions: * Jenkins 2.98 * Docker Plugin 1.1.2h2. Here are the steps to reproduce: * Create a *Cloud* configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method. Ensure that there is enough capacity (instance limits, etc) so that this is not a bottleneck. * Create a Pipeline job with the following code (adjust for image labels if needed): {code:java}def flavors = ['centos6','centos7','sles11','sles12','ubuntu']def steps = [:]flavors.each{def flavor -> steps[flavor] = {stage(flavor) { timeout(1) { echo "Allocating ${flavor}" node("${flavor}&") { sh "date" } }} }}timestamps { parallel steps}{code} * Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out. * Restart Jenkins. * Job will succeed again for a little while. (UPDATE: restart does not always help, issue sometimes begins immediately after) h2. Jenkins logs are attached:Job run when provisioning hangs: [^jenkins-log-provisioning-fail.log]Job run when provisioning succeeds: [^jenkins-log-provisioning-success.log] *Note* that while Jenkins is blocked trying to allocate a node, I can manually allocate one using docker command-line, proving that the actual docker infrastructure is not the problem.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov edited a comment on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally *able to reproduce this* in an isolated Jenkins installation running the following versions: * Jenkins 2.98 * Docker Plugin 1.1.2h2. Here are the steps to reproduce: * Create a *Cloud* configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method. Ensure that there is enough capacity (instance limits, etc) so that this is not a bottleneck. * Create a Pipeline job with the following code (adjust for image labels if needed): {code:java}def flavors = ['centos6','centos7','sles11','sles12','ubuntu']def steps = [:]flavors.each{def flavor -> steps[flavor] = {stage(flavor) { timeout(1) { echo "Allocating ${flavor}" node("${flavor}&") { sh "date" } }} }}timestamps { parallel steps}{code} * Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out. * Restart Jenkins. * Job will succeed again for a little while.h2. Jenkins logs are attached:Job run when provisioning hangs: [^jenkins-log-provisioning-fail.log]Job run when provisioning succeeds: [^jenkins-log-provisioning-success.log] *Note* that while Jenkins is blocked trying to allocate a node, I can manually allocate one using docker command-line, proving that the actual docker infrastructure is not the problem.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov edited a comment on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally *able to reproduce this* in an isolated Jenkins installation running the following versions: * Jenkins 2.98 * Docker Plugin 1.1.2h2. Here are the steps to reproduce: * Create a *Cloud* configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method. Ensure that there is enough capacity (instance limits, etc) so that this is not a bottleneck. * Create a Pipeline job with the following code (adjust for image labels if needed): {code:java}def flavors = ['centos6','centos7','sles11','sles12','ubuntu']def steps = [:]flavors.each{def flavor -> steps[flavor] = {stage(flavor) { timeout(1) { echo "Allocating ${flavor}" node("${flavor}&") { sh "date" } }} }}timestamps { parallel steps}{code} * Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out. * Restart Jenkins. * Job will succeed again for a little while.h2. Jenkins logs are attached:Job run when provisioning hangs: [^jenkins-log-provisioning-fail.log]Job run when provisioning succeeds: [^jenkins-log-provisioning-success.log]
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov edited a comment on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally *able to reproduce this* in an isolated Jenkins installation running the following versions: * Jenkins 2.98 * Docker Plugin 1.1.2h2. Here are the steps to reproduce: * Create a *Cloud* configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method. * Create a Pipeline job with the following code (adjust for image labels if needed): {code:java}def flavors = ['centos6','centos7','sles11','sles12','ubuntu']def steps = [:]flavors.each{def flavor -> steps[flavor] = {stage(flavor) { timeout(1) { echo "Allocating ${flavor}" node("${flavor}&") { sh "date" } }} }}timestamps { parallel steps}{code} * Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out. * Restart Jenkins. * Job will succeed again for a little while.h2. Logs Jenkins logs are attached: When Job run when provisioning hangs: [^jenkins-log-provisioning-fail.log] When Job run when provisioning succeeds: [^jenkins-log-provisioning-success.log]
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov edited a comment on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally *able to reproduce this* in an isolated Jenkins installation running the following versions: * Jenkins 2.98 * Docker Plugin 1.1.2h2. Here are the steps to reproduce: * Create a *Cloud* configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method. * Create a Pipeline job with the following code (adjust for image labels if needed): {code:java}def flavors = ['centos6','centos7','sles11','sles12','ubuntu']def steps = [:]flavors.each{def flavor -> steps[flavor] = {stage(flavor) { timeout(1) { echo "Allocating ${flavor}" node("${flavor}&") { sh "date" } }} }}timestamps { parallel steps}{code} * Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out. * Restart Jenkins. * Job will succeed again for a little while. h2. Logs are attached:When provisioning hangs: [^jenkins-log-provisioning-fail.log]When provisioning succeeds: [^jenkins-log-provisioning-success.log]
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Alexander Komarov updated an issue Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: Alexander Komarov Attachment: .jenkins-log-provisioning-fail.log.swp Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Alexander Komarov updated an issue Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: Alexander Komarov Attachment: .jenkins-log-provisioning-success.log.swp Attachment: .jenkins-log-provisioning-fail.log.swp Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Alexander Komarov updated an issue Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: Alexander Komarov Attachment: .jenkins-log-provisioning-success.log.swp Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title Alexander Komarov updated an issue Jenkins / JENKINS-48490 Intermittently slow docker provisioning with no errors Change By: Alexander Komarov Attachment: jenkins-log-provisioning-success.log Attachment: jenkins-log-provisioning-fail.log Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48490) Intermittently slow docker provisioning with no errors
Title: Message Title
Alexander Komarov commented on JENKINS-48490
Re: Intermittently slow docker provisioning with no errors
I was finally able to reproduce this in an isolated Jenkins installation running the following versions:
Jenkins 2.98
Docker Plugin 1.1.2
Here are the steps to reproduce:
Create a Cloud configuration with several docker image template definitions. Mine are custom built images, but the random nature of this issue suggests that this should happen with any images. Note: I am using JNLP but the problem also occurs with "Attached" method.
Create a Pipeline job with the following code (adjust for image labels if needed):
def flavors = ['centos6','centos7','sles11','sles12','ubuntu']
def steps = [:]
flavors.each{def flavor ->
steps[flavor] = {
stage(flavor) {
timeout(1) {
echo "Allocating ${flavor}"
node("${flavor}&") {
sh "date"
}
}
}
}
}
timestamps {
parallel steps
}
Run this job a few times. Job will succeed at first, but after a few successful runs one or more of the image flavors will start to time out.
Restart Jenkins.
Job will succeed again for a little while.
Add Comment
