[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Basil Crow commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. I am attempting to fix this flaw in jenkinsci/throttle-concurrent-builds-plugin#57. This PR has been merged, and the master branch of Throttle Concurrent Builds now uses FlowExecutionList to calculate the number of running Pipeline jobs, which should work around the issue described in this bug. I have yet to release a new version of Throttle Concurrent Builds with this fix, but there is an incremental build available here. Anna Tikhonova, are you interested in testing this incremental build before I do an official release? Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.183755.1500293058000.2104.1580946060306%40Atlassian.JIRA.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Basil Crow commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. Offhand this sounds like a flaw in TCB. I am attempting to fix this flaw in jenkinsci/throttle-concurrent-builds-plugin#57. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-45571 "likely stuck" job is not actually stuck. Change By: Vivek Pandey Labels: api triaged-2018-11 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Jesse Glick commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. TCP prevents scheduling more builds because it considers those hanging executors. Offhand this sounds like a flaw in TCB. This PR introduced that behavior, purportedly to support the build-flow plugin (a conceptual predecessor of Pipeline née Workflow). If TCB intends to throttle builds per se (rather than work done by those builds—typically node blocks for Pipeline), then there are more direct ways of doing this than counting Executor slots. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Devin Nusbaum edited a comment on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
[~atikhonova] Are you able upload the build directory of the build matching the stuck executor? Specifically, it would be helpful to see {{build.xml}} and the xml file(s) in the {{workflow}} directory. EDIT: I see now that you can't easily tell which are stuck and which are good. If you can find an executor with {{likelyStuck: true}}, and whose build looks like it has otherwise completed or is suck, that would be a great candidate.Another note: JENKINS- 52165 38381 will change the control flow here significantly.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Jesse Glick commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. gives you the jobs currently running This is not really an appropriate API query to use for that question. If your interest is limited to all Pipeline builds, FlowExecutionList is likely to be more useful. If you are looking at builds of a particular job (Pipeline or not), I think that information is available from the endpoint for that job. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Devin Nusbaum edited a comment on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
[~atikhonova] Are you able upload the build directory of the build matching the stuck executor? Specifically, it would be helpful to see {{build.xml}} and the xml file(s) in the {{workflow}} directory. EDIT: I see now that you can't easily tell which are stuck and which are good. If you can find an executor with {{likelyStuck: true}}, and whose build looks like it has otherwise completed or is suck, that would be a great candidate. Another note: JENKINS-52165 will change the control flow here significantly.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Devin Nusbaum edited a comment on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
[~atikhonova] Are you able upload the build directory of the build matching the stuck executor? Specifically, it would be helpful to see \ {{build.xml}} and the xml file(s) in the \ {{workflow}} directory. EDIT: I see now that you can't easily tell which are stuck and which are good. If you can find an executor with {{likelyStuck: true}}, and whose build looks like it has otherwise completed or is suck, that would be a great candidate.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Anna Tikhonova edited a comment on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
[~dnusbaum] unfortunately, I don't. I've got a few 1000+ LOC pipelines running continuously. I do not know how to tell with which one leaves executors and when.Pipeline build that has such "likelyStuck" executor looks completed on its build page (no progress bars, build status is set). But I still can see a matching OneOffExecutor on master:{code:java} "_class" : "hudson.model.Hudson$MasterComputer", "oneOffExecutors" : [{ "currentExecutable" : {"_class" : "org.jenkinsci.plugins.workflow.job.WorkflowRun","building" : false,// always false for these lost executors"result" : "SUCCESS",// always set to some valid build status != null"url" : "JENKINS/job/PIPELINE/BUILD_NUMBER/" }, "likelyStuck" : false// can be true or false}, ...{code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Devin Nusbaum commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. Anna Tikhonova Are you able upload the build directory of the build matching the stuck executor? Specifically, it would be helpful to see build.xml and the xml file(s) in the workflow directory. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Anna Tikhonova edited a comment on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
[~dnusbaum] unfortunately, I don't. I've got a few 1000+ LOC pipelines running continuously. I do not know how to tell with one leaves executors and when. Pipeline build that has such "likelyStuck" executor looks completed on its build page (no progress bars, build status is set). But I still can see a matching OneOffExecutor on master:{code:java} "_class" : "hudson.model.Hudson$MasterComputer", "oneOffExecutors" : [{ "currentExecutable" : {"_class" : "org.jenkinsci.plugins.workflow.job.WorkflowRun","building" : false,// always false for these lost executors"result" : "SUCCESS",// always set to some valid build status != null"url" : "JENKINS/job/PIPELINE/BUILD_NUMBER/" }, "likelyStuck" : false// can be true or false}, ...{code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Anna Tikhonova commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. Devin Nusbaum unfortunately, I don't. I've got a few 1000+ LOC pipelines running continuously. I do not know how to tell with one leaves executors and when. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Sam Van Oort commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. Note from investigation: so, separate from JENKINS-50199 there appears to be a different but related failure mode: 1. The symptoms described by Anna will be reproduced if the build completes (WorkflowRun#finish is called), but the copyLogsTask never gets invoked or fails, since that is what actually removes the FlyWeightTask and kills the OneOffExecutor. See the CopyLogsTask logic - https://github.com/jenkinsci/workflow-job-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java#L403 2. If the AsynchronousExecution is never completed, we'll see a "likelyStuck" executor for each OneOffExecutor Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title Devin Nusbaum commented on JENKINS-45571 Re: "likely stuck" job is not actually stuck. Anna Tikhonova The fact that you are seeing the issue without restarting Jenkins is very interesting. Do you have a pipeline which is able to reproduce the problem consistently? Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Anna Tikhonova commented on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
Why this bug could be of more interest is that it intervenes Throttle Concurrent Build plugin scheduling. TCP prevents scheduling more builds because it considers those hanging executors. Once there are more hanging executors than maximum total concurrent builds configured for a job (N), the job is forever stuck ("pending—Already running N builds across all nodes").
Add Comment
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-45571) "likely stuck" job is not actually stuck.
Title: Message Title
Anna Tikhonova commented on JENKINS-45571
Re: "likely stuck" job is not actually stuck.
I'm seeing this issue as well. Lots of executors listed in /computer/api/json?pretty=true=computer[oneOffExecutors[likelyStuck,currentExecutable[building,result,url]]]{0} in the following state:
{
"currentExecutable" : {
"_class" : "org.jenkinsci.plugins.workflow.job.WorkflowRun",
"building" : false,
"result" : "SUCCESS",
"url" : url
},
"likelyStuck" : true
}
However, in my case it doesn't seem to be related to resuming pipelines at Jenkins startup. I have written a script to cleanup such executors. Haven't restarted Jenkins since the script has run, and still I see the new executors like those.
Add Comment
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
