[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title
Michael Mrozek commented on JENKINS-53652
Re: InterruptedException during slave workspace groovy clean up pipeline
Vassilena Treneva: Without the loop you're preventing the exception from failing your build, but not actually deleting the directory you tried to delete with fp.deleteRecursive(); – the operation failed. I didn't have any cap on the number of retries; it seems like it deletes some files each time before getting interrupted (it takes about 5 seconds to get interrupted according to the step timer), so eventually it succeeds given enough attempts. I did this:
for(;;) {
try {
dir.deleteRecursive()
break
} catch(InterruptedException e) {
echo "Delete interrupted; retrying"
} catch(e) {
echo "Failed to remove directory: $e"
break
}
}
You could add a max number of attempts to the for loop if you wanted
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Jeff Thompson closed an issue as Done Jenkins / JENKINS-53652 InterruptedException during slave workspace groovy clean up pipeline Change By: Jeff Thompson Status: Open Closed Resolution: Done Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Jeff Thompson commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline I'm glad you were able to figure something out. I would feel a lot more comfortable if we understood what is going on, what is causing the InterruptedException. If you figure anything out on that, please share. But, at least you figured out how to get things working well. As for your question on killing processes, you would likely get better results by asking that in the Jenkins users forums. It looks like you've got something working so I'm going to go ahead and close this issue. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline A side question (hope I am not spoiling the issue) – is there a way to check and kill all remaining process in the workspace without using OS commands, but instead using Jenkins Groovy Api for this? Something like an alternative to kill -9 for all processes in the workspace… This check (!node.getComputer().isOffline()) is good for figuring out if the node is currently building, but if by any chance some process has been left alive, the script would eventually fail. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title
Vassilena Treneva commented on JENKINS-53652
Re: InterruptedException during slave workspace groovy clean up pipeline
Okay, so I think I do not need the loop at all! This script works: import hudson.model.Node import hudson.model.Slave import jenkins.model.Jenkins import hudson.FilePath; pipeline { triggers { cron('H 1 * * *') } agent { node { label 'master' } } options { timeout(time: 3, unit: 'HOURS') buildDiscarder(logRotator(daysToKeepStr: '30', numToKeepStr: '30', artifactDaysToKeepStr:'1', artifactNumToKeepStr: '30')) timestamps() } stages { stage ('Wipe Out Slave Workspaces') { steps { script { Jenkins jenkins = Jenkins.instance def jenkinsNodes = jenkins.nodes for (Node node in jenkinsNodes) { if (!node.getComputer().isOffline()) { if(node.getComputer().countBusy()==0) { FilePath fp = node.getRootPath().child("workspace"); println("Processing: " + node.getDisplayName()) println ("Wiping out: " + fp) def now = new Date() println now.format("-MM-dd HH:mm:ss", TimeZone.getTimeZone('GMT+2')) println("--") try { fp.deleteRecursive(); } catch (InterruptedException e) { println "InterruptedException caught!" e.printStackTrace(); } } } } } } } } } Michael Mrozek Many thanks again! Jeff Thompson Thank you for your detailed answers and ideas and most of all - for not ignoring my problem
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title
Vassilena Treneva commented on JENKINS-53652
Re: InterruptedException during slave workspace groovy clean up pipeline
Michael Mrozek, Many thanks for this hint! I will try to adjust my script in the same way. I will attempt something like this: int counter = 0; while (counter < 10) { try { fp.deleteRecursive(); } catch (InterruptedException e) { counter ++; e.printStackTrace(); } } I am not clear what the exit condition of the while should be? What is yours? Just some counter or some other condition? If you can share a snippet with me - that would be helpful.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Michael Mrozek commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline I have the same problem while doing the same kind of operation. I ended up just wrapping the deleteRecursive in a try/catch block to catch and ignore the InterruptedException, with a loop around that to keep trying until it works. It makes progress before getting interrupted each time, so eventually everything gets deleted, but for large directories it can get interrupted dozens of times Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Jeff Thompson commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline Hmm ... I've really got no idea. If you have access to the agent systems you could try looking on there manually and seeing if there is anything interesting in the remoting logs. I doubt you'll find anything. It looks like something else is interrupting the call. Something running on the master. Given your log with timestamps, it's not like this job is taking a long time. None of the threads look suspicious to me but I don't have much familiarity with them. Maybe someone with more experience in other areas, such as pipeline, may have other ideas but I don't know who to suggest. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline Hey, The job does not complete. It fails. I cannot see a pattern when it tends to fail. Sometimes it takes ~10 minutes, sometimes less. And it happens when working with different slaves I have added timestamps to the execution and I am attaching the console output with timestamps (https://issues.jenkins-ci.org/secure/attachment/44537/ConsoleOutputWithTimestamps.txt) as well as the threads I see immediately after the job has failed (https://issues.jenkins-ci.org/secure/attachment/44536/ThreadsAfterJobFailure.txt) I was also thinking if it makes sense that I log a bit more data about the threads from the pipeline itself? I am not sure however what is the context and how to access the current thread... Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva updated an issue Jenkins / JENKINS-53652 InterruptedException during slave workspace groovy clean up pipeline Change By: Vassilena Treneva Attachment: ThreadsAfterJobFailure.txt Attachment: ConsoleOutputWithTimestamps.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Jeff Thompson commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline It's hard to say. Some thread sends an interrupt to the thread waiting on the request. It doesn't look like it comes from Remoting so it must be one of the many other threads in that list. Does this job complete? Or does it terminate part way through? Any indication of how long different parts of it take? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline Attached a list of the active threads during the job execution (https://issues.jenkins-ci.org/secure/attachment/44535/ThreadsDuringTheRun.txt) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva updated an issue Jenkins / JENKINS-53652 InterruptedException during slave workspace groovy clean up pipeline Change By: Vassilena Treneva Attachment: ThreadsDuringTheRun.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline What kind of information would be helpful? It is 100% reproducible on our side, so I can try to collect whatever is needed... Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Jeff Thompson commented on JENKINS-53652 Re: InterruptedException during slave workspace groovy clean up pipeline There isn't enough information provided for me to be able to find any indication of what is going on from the Remoting side. Maybe the pipeline or some other component is interrupting the channel. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title
Vassilena Treneva updated an issue
Jenkins / JENKINS-53652
InterruptedException during slave workspace groovy clean up pipeline
Change By:
Vassilena Treneva
We use a declarative pipeline to wipe out all slaves workspaces (attached here - Jenkinsfile)It generally does some processing and attempts to delete the workspace like this:FilePath fp = node.getRootPath().child("workspace");fp.deleteRecursive(); It starts working well and fails with InterruptedException at some point: Started by timerLightweight checkout support not available, falling back to full checkout.Checking out svn http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace into /FS/fslocal/jenkinsWorkspace/jobs/admin_wipeout_workspaces/workspace@script to read JenkinsfileUpdating http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace at revision '2018-09-19T01:47:00.338 +0200' --quietAt revision 112786No changes for http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace since the previous buildRunning in Durability level: MAX_SURVIVABILITYLoading library templates@branches/dev/jenkins2_2/build/change-management/jenkinsfiles/templatesOpening connection to http://svndae.apama.com/um/Updating http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/templates@112786 at revision 112786At revision 112786No changes for http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/templates since the previous build[Pipeline] nodeRunning on Jenkins in /FS/fslocal/jenkinsWorkspace/jobs/admin_wipeout_workspaces/workspace[Pipeline] {[Pipeline] stage[Pipeline] { (Declarative: Checkout SCM)[Pipeline] checkoutUpdating http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace at revision '2018-09-19T01:47:00.338 +0200' --quietAt revision 112786No changes for http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace since the previous build[Pipeline] }[Pipeline] // stage[Pipeline] withEnv[Pipeline] {[Pipeline] stage[Pipeline] { (Wipe Out Slave Workspaces)[Pipeline] script[Pipeline] {[Pipeline] echoProcessing: daeosx109v04.eur.ad.sag[Pipeline] echoWiping out: /Users/nirdevadm/workspace/workspace[Pipeline] echo--[Pipeline] echoProcessing: daeosx109v05.eur.ad.sag[Pipeline] echoWiping out: /Users/nirdevadm/jenkinsWorkspace/workspace[Pipeline] echo--[Pipeline] echoProcessing: daeosx109v11.eur.ad.sag[Pipeline] echoWiping out: /Users/nirdevadm/workspace/workspace[Pipeline] echo--[Pipeline] echoProcessing: sofum01.eur.ad.sag[Pipeline] echoWiping out: /home/nirdevadm/jenkins/workspace[Pipeline] echo--[Pipeline] echoProcessing: umlinuxbuild10.eur.ad.sag[Pipeline] echoWiping out: /FS/fslocal/jenkins_slave/workspace[Pipeline]
[JIRA] (JENKINS-53652) InterruptedException during slave workspace groovy clean up pipeline
Title: Message Title Vassilena Treneva created an issue Jenkins / JENKINS-53652 InterruptedException during slave workspace groovy clean up pipeline Issue Type: Bug Assignee: Jeff Thompson Attachments: Jenkinsfile Components: pipeline, remoting Created: 2018-09-19 08:36 Environment: Jenkins ver. 2.140 Labels: remoting groovy-script pipeline slave Priority: Minor Reporter: Vassilena Treneva We use a declarative pipeline to wipe out all slaves workspaces (attached here - Jenkinsfile) It starts working well and fails with InterruptedException at some point: Started by timer Lightweight checkout support not available, falling back to full checkout. Checking out svn http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace into /FS/fslocal/jenkinsWorkspace/jobs/admin_wipeout_workspaces/workspace@script to read Jenkinsfile Updating http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace at revision '2018-09-19T01:47:00.338 +0200' --quiet At revision 112786 No changes for http://svndae.apama.com/um/branches/dev/jenkins2_2/build/change-management/jenkinsfiles/wipeout_slave_workspace since the previous build Running in Durability level: MAX_SURVIVABILITY Loading library templates@branches/dev/jenkins2_2/build/change-management/jenkinsfiles/templates Opening
