[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Alex Raber edited a comment on JENKINS-54540 Re: Pods stuck in error state is not cleaned up bumping this thread – I'm seeing this issue, and it seems to correlate to when the jenkins-master container is moved to another kubernetes node due to resource scaling. I think this issue can be resolved if jnlp takes in a timeout threshold to allow jnlp to run without failing if connection to master is lost for X seconds. Potentially this Apparently, in such an event, error containers are expected and they are essentially zombies that the master will not take care of. The master will however ensure that the jobs resume once master is related to JENKINS-44785 back online . Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.9088.1580424120289%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Alex Raber edited a comment on JENKINS-54540 Re: Pods stuck in error state is not cleaned up bumping this thread – I'm seeing this issue, and it seems to correlate to when the jenkins-master container is moved to another kubernetes node due to resource scaling. I think this issue can be resolved if jnlp takes in a timeout threshold to allow jnlp to run without failing if connection to master is lost for X seconds.Potentially this is related to JENKINS-44785. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.9000.1580413020292%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Alex Raber commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up bumping this thread – I'm seeing this issue, and it seems to correlate to the jenkins-master container is moved to another kubernetes node due to resource scaling. I think this issue can be resolved if jnlp takes in a timeout threshold to allow jnlp to run without failing if connection to master is lost for X seconds. Potentially this is related to JENKINS-44785. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.8978.1580412420380%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Michael Odell commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up We also saw the problem when podRetention was set to OnError (including the pods that were in Running and not Error) and switched to Never to try to get it to go away. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.6212.1572539760184%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Michael Odell commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up I see the same thing. We have podRetention set to never, but out of the probably hundreds of jobs we run per day, a handful of the pods stick around in an error state or in "Running" where only a subset of the containers are running (i.e. READY: 3/4 STATUS: Running). FWIW, our jobs don't reuse pods. I suppose if they did we might ( ? ) see this less often. Cleanup is not terribly onerous except that it has to be done every day. I have a hard time seeing how this can be classified as an enhancement rather than a bug. Given long enough, this will cause the system to stop working because of resource exhaustion. We happen to be on kubernetes 1.12.10, plugin version 1.18.3, and Jenkins version 2.176.3, but we also seen this with slightly older versions of all three, and I believe in testing with newer versions of Jenkins and plugin (but we're not running enough jobs on those newer versions to see it reliably). Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.6202.1572539520189%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Lu Shen commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up We see this issue as well when podRetention is set to Never. Kubernetes plugin version 1.12.3. In our case, we could get some 100 pods all in error and some pods were started at the same time. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.18386.1563834480140%40Atlassian.JIRA.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Jesse Glick assigned an issue to Unassigned Jenkins / JENKINS-54540 Pods stuck in error state is not cleaned up Change By: Jesse Glick Assignee: Carlos Sanchez Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.12913.1563306208692%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-54540 Pods stuck in error state is not cleaned up Change By: Jesse Glick Labels: jenkins jnlp jnlp-slave kuberentes plugin Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.26297.1560368160540%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-54540 Pods stuck in error state is not cleaned up Change By: Jesse Glick Labels: jenkins jnlp jnlp-slave kuberenetes-plugin kuberentes plugin Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.26242.1560368042772%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Rishi Thakkar commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up I see this issue as well when podRetention is set to never. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195312.1541679013000.24717.1560206820220%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Daniel Watrous commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up podRetention is set to never. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Carlos Sanchez commented on JENKINS-54540 Re: Pods stuck in error state is not cleaned up pods in error state are not cleaned by the plugin by default. Have you tried setting podRetention to never ? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Carlos Sanchez updated an issue Jenkins / JENKINS-54540 Pods stuck in error state is not cleaned up Change By: Carlos Sanchez Summary: Pods stuck in error state is not cleaned up Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54540) Pods stuck in error state is not cleaned up
Title: Message Title Carlos Sanchez updated an issue Jenkins / JENKINS-54540 Pods stuck in error state is not cleaned up Change By: Carlos Sanchez Issue Type: Bug Improvement Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
