[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber edited a comment on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar:{{for (aSlave in hudson.model.Hudson.instance.slaves) \{}}{{ if (aSlave.getComputer().isOffline()) \{}} } {{ println(aSlave.name + ' Deleted');}} } {{ aSlave.getComputer().doDoDelete();}} } {{ }}}{{ }}}From [ http://stackoverflow.com/questions/24072354/jenkins-is-there-a-way-to-remove-all-offline-nodes-slaves-batch-remove-nod ]
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber edited a comment on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar: \{code} {{ for (aSlave in hudson.model.Hudson.instance.slaves) \{ }} {{ if (aSlave.getComputer().isOffline()) \{ }} {{ println(aSlave.name + ' Deleted'); }} {{ aSlave.getComputer().doDoDelete(); }} {{ } } } \ { code { } }}From http://stackoverflow.com/questions/24072354/jenkins-is-there-a-way-to-remove-all-offline-nodes-slaves-batch-remove-nod
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber commented on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar: {code} for (aSlave in hudson.model.Hudson.instance.slaves) { if (aSlave.getComputer().isOffline()) { println(aSlave.name + ' Deleted'); aSlave.getComputer().doDoDelete(); } } {code}
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Hello, is there any update on this or an idea where to check? If there is anything we can do to help debug this or how to start looking to fix this would be great, we currently considering writing a script that cleans up these nodes, we may as well invest the effort in fixing it in the plugin =) a quick pointer of where to start looking would be great. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Damon Osgood commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails I am just now setting up Jenkins with Kubernetes slaves, and I do see this problem. I had thought it might have to do with failed builds, but extremely quick builds could also be the key. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails To clarify: The pods are terminated and deleted in kubernetes the jenkins slave reference of them in Jenkins is not deleted when they are terminated and deleted in kubernetes Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Carlos Sanchez commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Are you saying that the Pods don't get deleted in kubernetes, the jenkins slave is not deleted in Jenkins, or both ? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails I forgot: They get however cleaned up after a while, I think around 12h or so automatically. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Carlos Sanchez Sebastien Vas I have the same issue, on our side it does not seem to relate to a specific command but from what we gathered it happens when the job on the node (within a pipeline) takes less than 1-2 seconds. In that case the reference to the node does not seem to get cleaned from Jenkins. If the node actions we are scripting take longer it is cleanly removed from jenkins. It always get's removed from the kubernetes cluster when it ends in this state. Furthermore the build image we are using is rather large, around 1.5GB. On smaller ones we could not reproduce this behaviour. According to the Jenkins logs certain checks / actions are only done once a second, clould this be related to why the proper shutdown / kill of the pod is not realized within the plugin? I attached a screenshot of the situation. The log entries for all pods that end up like this follow the same pattern as below: Nov 10, 2016 12:13:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Created Pod: cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Waiting for Pod to be scheduled (0/100): cupenya-root-docker-1460271f3921 -- Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminating Kubernetes instance for slave cupenya-root-docker-1460271f3921 -- Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminated Kubernetes instance for slave cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Disconnected computer cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:50 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: Computer.threadPoolForRemoting [#138] for cupenya-root-docker-1460271f3921 terminated -- Nov 10, 2016 12:18:59 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:19:19 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:20:39 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:20:49 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:21:29 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- You can maybe try to reproduce it with this image as it is larger, in case that is the issue: cupenya/docker-jenkins-slave-cpy-root https://github.com/cupenya/docker-jenkins-slave-cpy-root/blob/master/Dockerfile I'm happy to provide more details or in case you can't reproduce this in your environment show you a live example when this happens.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber updated an issue Jenkins / JENKINS-35246 Kubernetes nodes not getting deleted when a pipeline job fails Change By: Elmar Weber Attachment: suspended_pods.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Sebastien Vas commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Hi Carlos. Sorry for the late response I was out of the country. the slaves are actually gone from the kubernetes cluster, they are just listed as offline / suspended in Jenkins and since we keep creating them, they pile up. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Carlos Sanchez commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails can't reproduce in GCE A kubectl describe and kubectl logs of the pod that stays running would help Sebastien Vas are you using OpenShift too? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Sebastien Vas commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails I am experiencing the same behavior. It is kind of frustrating to have to manually delete slaves. Restarting Jenkins does not resolve the issue. I am worried that the suspended slaves are still accounted for, or at least creating a slow memory leak. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
