[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Reinhold Füreder commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Devin Nusbaum I can confirm that your fix works really fine! Because – now some coughing and red face – I accidentally restarted Jenkins master without waiting for pipelines to complete (of course looking forward to JENKINS-60434): and there were some non-minor real world pipelines running... Just one of them failed due to JENKINS-49365... Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.11279.1576652943246%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart? Yes, although note that you can also cancel /safeRestart before the restart happens, and the logic change fixes that case too. I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details. Ok, great! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.6784.1576180741472%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Nick Jones commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Thanks Devin Nusbaum. So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart? I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.6716.1576180501571%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL Both of these situations use the /safeRestart URL behind the scenes, which puts Jenkins into the same state as "Prepare for shutdown", which prevents new builds from being started and causes Pipeline builds to pause. The difference between /safeRestart and "Prepare for shutdown" is that safeRestart will also automatically restart Jenkins once all non-Pipeline jobs have completed and all Pipeline jobs have been paused, whereas "Prepare for shutdown" does not actually restart Jenkins. Even before Pipeline: Groovy version 2.78, once Jenkins restarted due to /safeRestart, all Pipelines should have resumed automatically, and they should continue to have that behavior in Pipeline: Groovy 2.78. If your Pipelines are not resuming after the restart, please open a new ticket, including steps to reproduce the issue from scratch and any messages from your Jenkins logs or Pipeline build logs that seem relevant. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Nick Jones commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Devin Nusbaum could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL, as enabled by https://plugins.jenkins.io/saferestart)? Do running pipeline jobs get paused in those circumstances too and now (with Pipeline: Groovy 2.78) automatically resumed once Jenkins is back up? Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.6304.1576151462391%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Reinhold Füreder commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Thanks again Devin Nusbaum! And following your advice => JENKINS-60434 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.4879.1576045681428%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum edited a comment on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines A fix for this issue was just released in Pipeline: Groovy Plugin version 2.78. I think there is/was some confusion as to the expected behavior (myself included!), so let me try to clarify: When Jenkins prepares for shutdown, all running Pipelines are paused, and this is the intended behavior. The unintended behavior was that if you canceled shutdown, Pipelines remained paused. This has been fixed in 2.78; Pipelines will now resume execution if shutdown is canceled. Before 2.78, you had to manually pause and unpause each Pipeline to get it to resume execution, or restart Jenkins. Additionally, preparing Jenkins for shutdown and canceling shutdown now each cause a message to be printed to Pipeline build logs indicating that the Pipeline is being paused or resumed due to shutdown so that it is easier to understand what is happening. Based on comments here and elsewhere, I think some users would prefer a variant of "Prepare for shutdown" in which Pipelines continue executing to completion, the same as other types of jobs like Freestyle. If that is something you want, please open a new ticket, describing your use case and the desired behavior.For anyone curious as to why Pipelines are paused when Jenkins prepares for shutdown, instead of continuing to execute and only saving at the last possible second when Jenkins is stopped, the reasoning is to avoid race conditions saving Pipeline metadata that could prevent Pipelines from resuming correctly.If there is some other aspect of this issue that you would like to see addressed, or a different behavior you would prefer, please open a new ticket describing your particular use case.Thanks! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum updated JENKINS-34256 Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Devin Nusbaum Status: In Review Resolved Resolution: Fixed Released As: workflow-cps 2.78 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.4597.1576014544889%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines A fix for this issue was just released in Pipeline: Groovy Plugin version 2.78. I think there is/was some confusion as to the expected behavior (myself included!), so let me try to clarify: When Jenkins prepares for shutdown, all running Pipelines are paused, and this is the intended behavior. The unintended behavior was that if you canceled shutdown, Pipelines remained paused. This has been fixed in 2.78; Pipelines will now resume execution if shutdown is canceled. Before 2.78, you had to manually pause and unpause each Pipeline to get it to resume execution, or restart Jenkins. Based on comments here and elsewhere, I think some users would prefer a variant of "Prepare for shutdown" in which Pipelines continue executing to completion, the same as other types of jobs like Freestyle. If that is something you want, please open a new ticket, describing your use case and the desired behavior. For anyone curious as to why Pipelines are paused when Jenkins prepares for shutdown, instead of continuing to execute and only saving at the last possible second when Jenkins is stopped, the reasoning is to avoid race conditions saving Pipeline metadata that could prevent Pipelines from resuming correctly. If there is some other aspect of this issue that you would like to see addressed, or a different behavior you would prefer, please open a new ticket describing your particular use case. Thanks! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Reinhold Füreder commented on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
Devin Nusbaum Thanks, I think that ("The main change in my PR is that today, if you cancel shutdown after clicking "Prepare for shutdown", the Pipeline builds stay paused. You have to manually pause and unpause the builds to get them to resume or restart Jenkins. After my PR, canceling shutdown will unpause the builds and resume them automatically.") should actually really address one of my (many months ago) experienced problems in this concern! (Because I have a groovy init hook script that always configures Jenkins to start in so-called quiet mode...) As other users more or less diplomatically commented, there is still room for related important enhancements: maybe these can be collected and discussed and prioritised and addressed in another future sprint/story? (And I think Jenkins is still in massive use nowadays and hopefully not dead for a long time...)
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.1208.1575531361659%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Friedrich Clausen commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines As mentioned above, in the modern world of transient agents such as Kubernetes pods, this is quite painful. Transient agents are likely to become more prevelant. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.1040.1575507121314%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Carlton Brown commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines 4 years old serious usability issue, hasn't been fixed, won't be fixed because nobody cares. Jenkins is dead, use something else. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.973.1575505561367%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Ferruccio Bongianni I think the intention of the message is something like how you might use wall on a multi-user Unix system, in the sense that the message is a way for admins to signify to anyone that might be using Jenkins that it will be shut down at some point (just guessing, I did not add the feature). The admin still has to actually initiate the shutdown themselves, so for admins, I agree, the message is confusing. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.726.1575490981334%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Alex Varju commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines In the modern world of transient agents (e.g. Kubernetes pods) that won't exist after restart, this approach of pausing the pipeline is painful. It would sure be nice if there was a way to allow current jobs to finish while not allowing queued jobs to be started. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.660.1575490921480%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Ferruccio Bongianni commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Hi Devin, thanks for your prompt reply. It is very confusing indeed. Shouldn't the message in red saying something different than 'Jenkins is going to shut down' if that is not true (because it actually does not shut down at all, ever). Also, I haven't tried it right now, but I'm fairly confident this is exactly what happens when you update a plugin that needs Jenkins to be restarted. if you've got pipelines running it will never do until you kill those pipelines. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.593.1575490801454%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Ferruccio Bongianni Yes, it's confusing, but that's the intended behavior. Clicking "Prepare for shutdown" pauses all running Pipelines. My PR prints a message to the build log of Pipelines when this happens to make it clear that the build is paused. Once the build is paused, Jenkins can be restarted, and the Pipeline will resume after the restart. If you are only using "Prepare for shutdown" to restart Jenkins without breaking in-progress builds of non-Pipeline jobs, you can navigate to the /safeRestart URL, which is like "Prepare for shutdown" except that it automatically restarts after all non-Pipeline jobs complete. The Pipeline builds should resume after Jenkins restarts with or without my PR. The main change in my PR is that today, if you cancel shutdown after clicking "Prepare for shutdown", the Pipeline builds stay paused. You have to manually pause and unpause the builds to get them to resume or restart Jenkins. After my PR, canceling shutdown will unpause the builds and resume them automatically. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.584.1575490323935%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Ferruccio Bongianni edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
I am experiencing the same problem as described in the title. This is how to I reproduce it: * Run an instance of jenkins/jenkins:lts image (which for me has version [Jenkins ver. 2.190.2|https://jenkins.io/]) * create a simple pipeline: ** {code:java}pipeline {agent anystages {stage('x') {steps {sh 'sleep 30'sh 'sleep 30'}}}} {code} * run a build of the pipeline above, then go to {{Manage Jenkins}} and click on {{Prepare for shutdown}} At this point Jenkins shows the red stripe {{Jenkins is going to shut down}} but it never does. The pipeline never proceed (in fact, it doesn't even reach the second {{sh}} if the prepare for shutdown happened during the first sleep. The pipeline hangs, and never terminate.I am currently at Jenkins world and showed the behaviour to [~bitwiseman] today.
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.381.1575489963458%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Ferruccio Bongianni edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
I am experiencing the same problem as described in the title. This is how I reproduce it: * Run an instance of a container from jenkins/jenkins:lts image (which for me has version [Jenkins ver. 2.190.2|https://jenkins.io/]) * create a simple pipeline: ** {code:java}pipeline {agent anystages {stage('x') {steps {sh 'sleep 30'sh 'sleep 30'}}}} {code} * run a build of the pipeline above, then go to {{Manage Jenkins}} and click on {{Prepare for shutdown}} At this point Jenkins shows the red stripe {{Jenkins is going to shut down}} but it never does. The pipeline never proceed (in fact, it doesn't even reach the second {{sh}} if the prepare for shutdown happened during the first sleep. The pipeline hangs, and never terminate.I am currently at Jenkins world and showed the behaviour to [~bitwiseman] today.
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.448.1575489964326%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
ferruccio bongianni commented on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
I am experiencing the same problem as described in the title. This is how to reproduce it:
Run an instance of jenkins/jenkins:lts image (which for me has version Jenkins ver. 2.190.2)
create a simple pipeline:
pipeline {
agent any
stages {
stage('x') {
steps {
sh 'sleep 30'
sh 'sleep 30'
}
}
}
}
run a build of the pipeline above, then go to Manage Jenkins and click on Prepare for shutdown
At this point Jenkins shows the red stripe Jenkins is going to shut down but it never does. The pipeline never proceed (in fact, it doesn't even reach the second sh if the prepare for shutdown happened during the first sleep. The pipeline hangs, and never terminate. I am currently at Jenkins world and showed the behaviour to Liam Newman today.
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Devin Nusbaum edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
I am looking into this issue in [PR 340| [ https://github.com/jenkinsci/workflow-cps-plugin/pull/340] ] . I am a little confused by some of the comments in this thread. As far as I can tell, this has never worked, regardless of what step is executing when quiet mode is enabled, because there is no code to tell Pipeline executions that quiet mode was cancelled and they should try to resume themselves. Maybe I am misunderstanding something or there are multiple distinct issues being discussed in the ticket, so I am going to reread all of the comments in case and do some additional testing with the {{sh}} step.
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.10695.1575324490641%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum updated JENKINS-34256 Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Devin Nusbaum Status: In Progress Review Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.10697.1575324490662%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum assigned an issue to Devin Nusbaum Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Devin Nusbaum Assignee: Devin Nusbaum Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.10628.1575324489807%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Devin Nusbaum commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines I am looking into this issue in [PR 340|https://github.com/jenkinsci/workflow-cps-plugin/pull/340]. I am a little confused by some of the comments in this thread. As far as I can tell, this has never worked, regardless of what step is executing when quiet mode is enabled, because there is no code to tell Pipeline executions that quiet mode was cancelled and they should try to resume themselves. Maybe I am misunderstanding something or there are multiple distinct issues being discussed in the ticket, so I am going to reread all of the comments in case and do some additional testing with the sh step. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.10561.1575324488861%40Atlassian.JIRA.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Chris Frolik commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines It is distressing that major issues like this sit for 3+ years with no resolution – and worse, not assigned to anyone. Since pipeline jobs are "the norm" now, the "Prepare For Shutdown" button is a trap for users to get their system into a broken state. If this issue cannot be fixed, at the very least there should be a warning label next to that button. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.169794.1460663849000.11290.1563203821580%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Vladimir K commented on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
According to my observations, the bug only affects Jenkins pipelines and happens when a Jenkins is put into shutdown mode when there are some pipelines running on background; those pipelines will not be able to proceed to next pipeline stage(s) and will indefinitely stuck in whatever last stage there was prior shutdown mode. This can be reproduced with the following pseudo-pipeline:
stages {
stage('build') {
steps {
sh('make build')
}
}
stage('prepare') {
steps {
//During this stage, Jenkins is put into shutdown mode
sh('make prepare-for-restart')
}
}
stage('deploy') {
steps {
// Pipeline will not reach this stage
sh('make deploy')
}
}
}
post {
always {
sh('echo Test')
}
}
The following pipeline will never reach neither deploy or post stages My guess shutdown prevents any new build threads to be executed and since each stage runs in separate thread (for serialization purposes), pipelines get stuck. This behavior seems to be intended because this allows Jenkins to continue stages after hard restart. In my use-case I would like to conduct a safe, controlled Jenkins restart, allowing any existing workloads to finish.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options,
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title awk space commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Having the same issue here on Jenkins 2.165 - even with simple 'sh "sleep 60"' test jobs. Attempting to work around the issue by checking "Do not allow the pipeline to resume after master restarts" and changing the pipeline to PERFORMANCE_OPTIMIZED makes the pipeline attempt to resume after restart (??) and makes me run into JENKINS-50407 instead. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Laszlo Gaal commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines We are also seeing this on LTS 2.150.2 quite regularly. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Torben Knerr edited a comment on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines same Same issue here with Jenkins LTS 2.150.2 I'm seeing this with pipeline durability set to "PERFORMANCE_OPTIMIZED" in the global configuration.[~svanoort] re-reading your comment here https://issues.jenkins-ci.org/browse/JENKINS-34256?focusedCommentId=332080=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-332080 I am wondering why the currently executing pipeline should actually halt by design -- wouldn't it be more intuitive if any running pipelines just complete (as it was the case with freestyle jobs earlier) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Torben Knerr commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines same issue here with Jenkins LTS 2.150.2 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Hamid Gholami commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Any update? We have same issue on Jenkins. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Ali Narenji commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines We have the same problem on Jenkins 2.138.2. Is there any time estimation for resolving the issue? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Vivek Pandey Labels: 1.651.1 2.0 2.0-rc lts testfest triaged-2018-11 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Jose Blas Camacho Taboada assigned an issue to Unassigned Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Jose Blas Camacho Taboada Assignee: Jose Blas Camacho Taboada Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Josiah Eubank commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Also experiencing this new since around 2.140 in all pipeline jobs. These jobs are on lowest durability setting. Previously had no issue holding the queue by "preparing for shutdown" and the currently running jobs would finish. Now have to force Jenkins to restart to get rid of the jobs. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Mike Kozell That specific case sounds a lot like a gremlin we've been chasing on and off for quite a while. I'm assigning this to Jose Blas Camacho Taboada to investigate. I think what you report may be independent of what was discussed here though which is probably the root cause of the issue: https://issues.jenkins-ci.org/browse/JENKINS-34256?focusedCommentId=336282=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-336282 through https://issues.jenkins-ci.org/browse/JENKINS-34256?focusedCommentId=332080=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-332080 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort assigned an issue to Jose Blas Camacho Taboada Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Sam Van Oort Assignee: Sam Van Oort Jose Blas Camacho Taboada Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Mike Kozell commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Sam Van Oort we just came across this issue. Using the latest Jenkins pipeline plugins, we had a few shared groovy pipeline builds running when I clicked the "Prepare for shutdown" button. The running pipeline builds were running "steps.sh" commands. After waiting a long time, I logged into the Jenkins agents and discovered the processes had finished and the builds were just sitting there. I waited longer and finally clicked the pipeline "pause / resume" button twice which failed the build. I reviewed the Jenkins agents and didn't find anything unusual. The commands running inside the steps.sh() seemed to exit without issue. I checked the jenkins.log file on the master and didn't see any errors until I clicked pause / resume. Unfortunately this issue prevents us from being able to gracefully shutdown a busy Jenkins master to perform maintenance, upgrades, or even decommission. Below is the stack trace from the Jenkins log file after pause / resume was pressed on the hung builds. Sep 10, 2018 9:00:24 PM org.jenkinsci.plugins.workflow.cps.CpsStepContext completed WARNING: already completed CpsStepContext[14:timeout]:OwnerJenkins-Job-Name/1:Jenkins-Job-Name #1 java.lang.IllegalStateException: delivered here at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:353) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:333) at org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.onSuccess(CpsBodySubContext.java:68) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:119) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:366) at com.cloudbees.groovy.cps.impl.ValueBoundContinuation.receive(ValueBoundContinuation.java:21) at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:166) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Henti Smith commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines We're having similar issues. We use pipeline extensively to build on different platforms and types of slaves and we're also seeing the pipeline jobs finish but not remove from slave. Restarting jenkins, which is usually the reason for shutdown, gets the jobs even more out of shape as the pipeline job reconnects to the slave, then tries to continue on the slave, but cannot as it's waiting for executor on the slave it's running on. 21:13:21 Running on ella in /home/jenkins/slave/workspace/Security/SAMATE/SAMATE-java -- stuff happens here -- -- put jeckins in shutdown mode -- Waiting to resume part of Security » SALADE » SALADE-java #490: Jenkins is about to shut down -- Restart jenkins -- Resuming build at Tue Aug 21 07:50:31 BST 2018 after Jenkins restart Waiting to resume part of Security » SALADE » SALADE-java #490: Waiting for next available executor on ella My expectation is the pipeline job on that node would finish and the next pipeline job will be queued unassigned to a node to allow restart and connecting to a new node ? Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Tomasz Śniatowski commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines So with pipelines, what is the recommended way of completely stopping a busy Jenkins instance for maintenance? The maintenance is in part due to a broken pipeline resume a'la JENKINS-50199, so I specifically don't want any additional half-done pipelines waiting to be resumed. I also would prefer to avoid having to abort jobs. In JENKINS-38316 there's an explicit mention that "prepare for shutdown" is not that: The whole idea of "Prepare for shutdown" is to [...] allow you to finish currently running freestyle (Maven, matrix, …) builds. So if you /safeRestart Jenkins will restart as soon as any of those are completed, and running Pipeline builds will be left alone. What should I do then? Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Reinhold Füreder commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Sam Van Oort OK => JENKINS-51215 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Reinhold Füreder I would open a separate issue for Timeouts suspending executions – especially if you can come up with a consistent way to reproduce it. I saw it from time to time with Pipelines doing very complex processing (where we can't block the shutdown forever and shouldn't). My suspicion is that there's a subtle bug around the halt-at-shutdown logic, which may have been pre-existing but is visible now because the process is more closely monitored and logged now (also because we actually have some test coverage for it). Unfortunately By the way, you will sometimes be able to resume Pipelines after going into prepare-for-shutdown if the toggle happens at the right time – but in general there's no wakeup hook to resume execution (see notes above about how we plan to add one). Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Reinhold Füreder edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
More or less accidentally I just {color:#008000}successfully resumed 3 pipelines after cancelling the shutdown mode{color} after restarting Jenkins after putting Jenkins in shutdown mode (cf. also -JENKINS-38316-): * {color:#008000}and I did NOT have to wake them up manually by the "pause"-"resume" workaround{color} * maybe/presumably because I entered/started the shutdown mode in the middle of 'sh' steps, AND then waited until the end of this steps before restarting Jenkins?However, {color:red} the following minor (?) issues popped up {color} – please mind that there were actually two Jenkins restarts, because Jenkins Plugins were updated (just one in fact) after the first Jenkins restart via Jenkins init.d hook scripts, followed by a second restart (after the updates): # Resuming after Jenkins restart is slow{noformat}...Resuming build at Fri May 04 07:52:09 CEST 2018 after Jenkins restartWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ???Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downResuming build at Fri May 04 07:55:41 CEST 2018 after Jenkins restartWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ???Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downReady to run at Fri May 04 07:57:42 CEST 2018[Pipeline] sh07:57:42 [ACME] Running shell script...{noformat} # * * ... maybe because of running into timeouts when suspending (the more or less still suspended pipelines; because first action in init.d hook scripts is setting Jenkins in shutdown mode)!?{noformat}2018-05-04 07:52:09 INFO [hudson.WebAppMain$3 run] Jenkins is fully up and running2018-05-04 07:52:10 SEVERE [jenkins.model.Jenkins$24 run] Restarting VM as requested by SYSTEM2018-05-04 07:52:10 INFO [jenkins.model.Jenkins
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Reinhold Füreder edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
More or less accidentally I just {color: green #008000 }successfully resumed 3 pipelines after cancelling the shutdown mode{color} after restarting Jenkins after putting Jenkins in shutdown mode (cf. also - JENKINS-38316 - ):* {color: green #008000 }and I did NOT have to wake them up manually by the "pause"-"resume" workaround{color}* maybe/presumably because I entered/started the shutdown mode in the middle of 'sh' steps, AND then waited until the end of this steps before restarting Jenkins?However, the following minor ( \ ?) issue issues popped up -- – please mind that there were actually two Jenkins restarts, because Jenkins Plugins were updated (just one in fact) after the first Jenkins restart via Jenkins init.d hook scripts, followed by a second restart (after the updates): * # Resuming after Jenkins restart is slow{noformat}...Resuming build at Fri May 04 07:52:09 CEST 2018 after Jenkins restartWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ???Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downResuming build at Fri May 04 07:55:41 CEST 2018 after Jenkins restartWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ???Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downWaiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut downReady to run at Fri May 04 07:57:42 CEST 2018[Pipeline] sh07:57:42 [ACME] Running shell script...{noformat} * * ... maybe because of running into timeouts when suspending (the more or less still suspended pipelines; because first action in init.d hook scripts is setting Jenkins in shutdown mode)!?{noformat}2018-05-04 07:52:09 INFO [hudson.WebAppMain$3 run] Jenkins is fully up and running2018-05-04 07:52:10 SEVERE [jenkins.model.Jenkins$24 run] Restarting VM as requested by SYSTEM2018-05-04 07:52:10 INFO
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Reinhold Füreder commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines More or less accidentally I just successfully resumed 3 pipelines after cancelling the shutdown mode after restarting Jenkins after putting Jenkins in shutdown mode (cf. also JENKINS-38316): and I did NOT have to wake them up manually by the "pause"-"resume" workaround maybe/presumably because I entered/started the shutdown mode in the middle of 'sh' steps, AND then waited until the end of this steps before restarting Jenkins? However, the following minor (?) issue popped up – please mind that there were actually two Jenkins restarts, because Jenkins Plugins were updated (just one in fact) after the first Jenkins restart via Jenkins init.d hook scripts, followed by a second restart (after the updates): Resuming after Jenkins restart is slow ... Resuming build at Fri May 04 07:52:09 CEST 2018 after Jenkins restart Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ??? Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Resuming build at Fri May 04 07:55:41 CEST 2018 after Jenkins restart Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: ??? Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Waiting to resume part of Unattended-Upgrades » ACME 20180504-074900-revUNKNOWN: Jenkins is about to shut down Ready to run at Fri May 04 07:57:42 CEST 2018
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort edited a comment on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Jglick: so my suggested test case would first quiet down, then do something to wake up the program, then cancel quiet downJesse Glick·11:21 AM you might need to do something else there, b/c I suspect there is still a race condition in that test—`SemaphoreStep.succeed` will post a task to the CPS VM thread, but you need to wait for that task to actually be processed.Jesse Glick·11:21 AM *That* might be a valid use of `waitForSuspension`. [11:23 AM] Jesse Glick: @sam I would suggest that https://github.com/jenkinsci/workflow-cps-plugin/blob/564a12c05eb54d5a84062cd3bf1d68deb47e1d9f/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java#L236 if the reason for pausing is quiet down mode, you print something to the build log. That is something the test could `waitForMessage` to see.[11:23 AM] Jesse Glick: (as well as being better UX)[11:23 AM] Jesse Glick: for the other reason we already print a message to the log: https://github.com/jenkinsci/workflow-cps-plugin/blob/564a12c05eb54d5a84062cd3bf1d68deb47e1d9f/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1491[11:25 AM] Jesse Glick: thus the test would be: wait for `semaphore` step to start; set Jenkins to quiet mode; permit the step to finish; wait for the message saying that the build is paused due to quiet mode; cancel quiet mode; wait for build to complete on its own[11:26 AM] Jesse Glick: @Sam ^^^ Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort edited a comment on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Jglick: so my suggested test case would first quiet down, then do something to wake up the program, then cancel quiet down Jesse Glick·11:21 AM you might need to do something else there, b/c I suspect there is still a race condition in that test—`SemaphoreStep.succeed` will post a task to the CPS VM thread, but you need to wait for that task to actually be processed.Jesse Glick·11:21 AM *That* might be a valid use of `waitForSuspension`. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines [11:18 AM] Jesse Glick: @Sam I suspect you are quieting down in the middle of a `sleep`, then canceling that before anything else happens in the program, so… `CpsFlowExecution` never even notices that the state flipped [11:19 AM] Jesse Glick: @Sam JENKINS-38316 is about the more likely scenario that the admin goes into quiet down mode, then the CPS VM thread wakes up for whatever reason, sees that it is supposed to be in quiet mode, pauses, and then never receives a notification to do anything else (unless perhaps someone manually pauses and resumes the build) Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Jglick: so my suggested test case would first quiet down, then do something to wake up the program, then cancel quiet down Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Jesse's suggestions given issues with the testase here: [11:11 AM] Jesse Glick: very roughly: `semaphore 'wait'` can be the whole program; wait for it to start; `doQuietDown`; succeed step; `doCancelQuietDown`; wait for finish [11:11 AM] Jesse Glick: (maybe?) [11:11 AM] Jesse Glick: no need for `node`, `Thread.sleep`, or `waitForSuspension` [11:12 AM] Jesse Glick: untested, obviously, but I would try something along those lines Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort edited a comment on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Jesse's suggestions given issues with the testase testcase here: [11:11 AM] Jesse Glick: very roughly: `semaphore 'wait'` can be the whole program; wait for it to start; `doQuietDown`; succeed step; `doCancelQuietDown`; wait for finish[11:11 AM] Jesse Glick: (maybe?)[11:11 AM] Jesse Glick: no need for `node`, `Thread.sleep`, or `waitForSuspension`[11:12 AM] Jesse Glick: untested, obviously, but I would try something along those lines Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines See JENKINS-38316 for the same issue but with additional comments/info. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Sam Van Oort Component/s: workflow-cps-plugin Component/s: core Component/s: pipeline Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Reinhold Füreder I'm 99% sure this is where the hang originates from: https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java#L235 This behavior seemingly was is by design, because we wanted Pipelines to halt where they are (rather than completing fully) before shutdown. A better design might have been a separate "paused" state. Unfortunately AFAIK there's not a listener in Core that we can use to notify the Pipeline to wake back up when leaving QuietingDown mode. My best notion has been for halted Pipelines to poll periodically to see if we've left quietDown mode and then resume if so – doable but rather unfortunate. Worse, I can't seem to actually reproduce this behavior in unit test for reasons I'm still trying to ascertain, even though it's easy to demonstrate on a normal instance: https://github.com/svanoort/workflow-cps-plugin/blob/60308b567d4bff6904d6fbc3cb57fbda564eaff7/src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java#L249 Jesse Glick Do you have any notions here? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Reinhold Füreder commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Maybe JENKINS-38316 and in particular https://issues.jenkins-ci.org/browse/JENKINS-38316?focusedCommentId=332021=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-332021 explains the current situation (in case it has changed over the past almost 2 years)... Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Sam Van Oort edited a comment on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
It looks like I can replace Edit: REPRODUCE this locally like so: {{stage ("going to bed") \{}} \ { \ { node \{}} \ { \ { echo 'running a sleep'}} \ { \ { sh 'for i in `seq 1 70`; do echo "sleep $i" && sleep 1; done'}} \ { \ { } }}{{}}}Which means it should be debuggable/fixable now.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort updated JENKINS-34256 Jenkins / JENKINS-34256 Preparing Jenkins For Shutdown Hangs Running Pipelines Change By: Sam Van Oort Status: Reopened Open Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Sam Van Oort started work on JENKINS-34256 Change By: Sam Van Oort Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title
Sam Van Oort commented on JENKINS-34256
Re: Preparing Jenkins For Shutdown Hangs Running Pipelines
It looks like I can replace this locally like so: stage ("going to bed") { {{ node {}} {{ echo 'running a sleep'}} {{ sh 'for i in `seq 1 70`; do echo "sleep $i" && sleep 1; done'}} {{ } }} } Which means it should be debuggable/fixable now.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Luke Swithenbank commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines I seem to be having this same issue again with Jenkins 2.18, and the workflow-job plugin 2.5. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
