Thank you both! I particularly like the idea of just deleting before the job instead of after. I managed to get something else working too, though I think it could be masking the problem. I now run attrib -r -s /s /d before deleting (before the run and after, just in case). Seems to be pretty quick?
A run with this: http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55 full Jenkinsfile: https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/Jenkinsfile Thanks again! On Sunday, March 18, 2018 at 2:43:56 AM UTC-4, benjamin.a.lau wrote: > > We've used scripting around the sysinternals handle tool[1] to make > sure that everything which is touching files in the workspace is > actually terminated before trying to delete the workspace. We were > having more issues with this when trying to delete workspaces at the > ends of jobs so in many cases we just let it be and only delete the > workspace on the next run. > > Cheers, > Ben > > [1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle > > On Sat, Mar 17, 2018 at 3:15 PM, Christian Gagneraud <[email protected] > <javascript:>> wrote: > > On 18 March 2018 at 03:21, Sean Talts <[email protected] > <javascript:>> wrote: > >> Hey all, > >> > >> I'm having this sudden crazy problem where my Windows agent can't > delete > >> some files in its workspace anymore, failing all builds. It's getting a > >> java.nio.file.AccessDeniedException on some files that it created. You > can > >> see the full exceptions at the end of the log here: > >> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console > also > >> copied below. > >> > >> I've tried so many things, including switching JVMs on master and > agent, > >> upgrading and switching to LTS, uninstalling and reinstalling a new > agent > >> service in a new working directory… Any tips or advice would be > extremely > >> appreciated; been trying to solve this all day and night since it > started > >> Thursday when the master rebooted during a job on the Windows agent. > > > > We have thins kind of errors on Windows when there are still running > > processes that have a file descriptor opened. > > You could log on to the machine and check if there are any "zombie" > processes. > > > > Chris > > > >> > >> Thanks in advance, > >> Sean > >> > >> Full error: > >> > >> java.nio.file.AccessDeniedException: > >> > C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only > > > >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) > >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > >> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) > >> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) > >> at java.nio.file.Files.deleteIfExists(Unknown Source) > >> at hudson.Util.tryOnceDeleteFile(Util.java:297) > >> at hudson.Util.deleteFile(Util.java:253) > >> Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to > >> JNLP4-connect connection from > >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 > >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) > >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) > >> at hudson.remoting.Channel.call(Channel.java:952) > >> at hudson.FilePath.act(FilePath.java:998) > >> at hudson.FilePath.act(FilePath.java:987) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1192) > >> at > >> > org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) > > > >> at > >> > org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) > > > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) > > > >> at hudson.security.ACL.impersonate(ACL.java:290) > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) > > > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > >> at java.lang.Thread.run(Thread.java:748) > >> Caused: java.io.IOException: Unable to delete > >> > 'C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only'. > > > >> Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. > >> at hudson.Util.deleteFile(Util.java:258) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1225) > >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) > >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) > >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) > >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) > >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) > >> at hudson.FilePath.access$1100(FilePath.java:208) > >> at hudson.FilePath$13.invoke(FilePath.java:1195) > >> at hudson.FilePath$13.invoke(FilePath.java:1192) > >> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) > >> at hudson.remoting.UserRequest.perform(UserRequest.java:210) > >> at hudson.remoting.UserRequest.perform(UserRequest.java:53) > >> at hudson.remoting.Request$2.run(Request.java:364) > >> at > >> > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > > > >> at java.util.concurrent.FutureTask.run(Unknown Source) > >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> at hudson.remoting.Engine$1$1.run(Engine.java:94) > >> at java.lang.Thread.run(Unknown Source) > >> Also: java.nio.file.AccessDeniedException: > >> > C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\stanc_helper.stan > > > >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) > >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > >> at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source) > >> at java.nio.file.spi.FileSystemProvider.newOutputStream(Unknown Source) > >> at java.nio.file.Files.newOutputStream(Unknown Source) > >> at hudson.util.IOUtils.copy(IOUtils.java:42) > >> at hudson.FilePath.readFromTar(FilePath.java:2363) > >> Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to > >> JNLP4-connect connection from > >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 > >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) > >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) > >> at hudson.remoting.Channel.call(Channel.java:952) > >> at hudson.FilePath.act(FilePath.java:998) > >> at hudson.FilePath.act(FilePath.java:987) > >> at hudson.FilePath.untar(FilePath.java:540) > >> at > >> > org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) > > > >> at > >> > org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) > > > >> at > >> > org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) > > > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) > > > >> at hudson.security.ACL.impersonate(ACL.java:290) > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) > > > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > >> at java.lang.Thread.run(Thread.java:748) > >> Caused: java.io.IOException: Failed to extract StanSetup.tar.gz > >> at hudson.FilePath.readFromTar(FilePath.java:2373) > >> at hudson.FilePath.access$400(FilePath.java:208) > >> at hudson.FilePath$4.invoke(FilePath.java:542) > >> at hudson.FilePath$4.invoke(FilePath.java:540) > >> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) > >> at hudson.remoting.UserRequest.perform(UserRequest.java:210) > >> at hudson.remoting.UserRequest.perform(UserRequest.java:53) > >> at hudson.remoting.Request$2.run(Request.java:364) > >> at > >> > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > > > >> at java.util.concurrent.FutureTask.run(Unknown Source) > >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> at hudson.remoting.Engine$1$1.run(Engine.java:94) > >> at java.lang.Thread.run(Unknown Source) > >> Caused: java.io.IOException: remote file operation failed: > >> > C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA > > > >> at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from > >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 > >> at hudson.FilePath.act(FilePath.java:1005) > >> at hudson.FilePath.act(FilePath.java:987) > >> at hudson.FilePath.untar(FilePath.java:540) > >> at > >> > org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) > > > >> at > >> > org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) > > > >> at > >> > org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) > > > >> Caused: java.io.IOException: remote file operation failed: > >> > C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA > > > >> at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from > >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 > >> at hudson.FilePath.act(FilePath.java:1005) > >> at hudson.FilePath.act(FilePath.java:987) > >> at hudson.FilePath.deleteRecursive(FilePath.java:1192) > >> at > >> > org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) > > > >> at > >> > org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) > > > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) > > > >> at hudson.security.ACL.impersonate(ACL.java:290) > >> at > >> > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) > > > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > >> at java.lang.Thread.run(Thread.java:748) > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Jenkins Users" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected] <javascript:>. > >> To view this discussion on the web visit > >> > https://groups.google.com/d/msgid/jenkinsci-users/6eaa7b8d-d69b-4c5e-a818-2a94643ce96d%40googlegroups.com. > > > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups "Jenkins Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTjt-yWgMg58S%3DQNV7wy9r84J4hr270rjqTU-ck0sps%2BpA%40mail.gmail.com. > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3fce1831-0a1f-4a66-82d1-6180b5718b98%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
