On Sun, Mar 18, 2018 at 5:40 PM Sean Talts <[email protected]> wrote:
> Thank you both! I particularly like the idea of just deleting before the > job instead of after. I managed to get something else working too, though I > think it could be masking the problem. I now run attrib -r -s /s /d before > deleting (before the run and after, just in case). Seems to be pretty quick? > > A run with this: > http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55 > full Jenkinsfile: > https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/Jenkinsfile > > Thanks again! > > If you're willing to use an "attrib" command, then you might also consider "robocopy /mir" from an empty directory to the workspace directory. I had problems deleting directories when they contained a directory structure that was too long. Using "robocopy /mir" from an empty directory to the destination directory was able to delete those too long files when the Windows "rmdir" command would not. I preferred that solution in my case because Robocopy is a standard part of Windows 7 and Windows 10. Mark Waite > On Sunday, March 18, 2018 at 2:43:56 AM UTC-4, benjamin.a.lau wrote: >> >> We've used scripting around the sysinternals handle tool[1] to make >> sure that everything which is touching files in the workspace is >> actually terminated before trying to delete the workspace. We were >> having more issues with this when trying to delete workspaces at the >> ends of jobs so in many cases we just let it be and only delete the >> workspace on the next run. >> >> Cheers, >> Ben >> >> [1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle >> >> On Sat, Mar 17, 2018 at 3:15 PM, Christian Gagneraud <[email protected]> >> wrote: >> > On 18 March 2018 at 03:21, Sean Talts <[email protected]> wrote: >> >> Hey all, >> >> >> >> I'm having this sudden crazy problem where my Windows agent can't >> delete >> >> some files in its workspace anymore, failing all builds. It's getting >> a >> >> java.nio.file.AccessDeniedException on some files that it created. You >> can >> >> see the full exceptions at the end of the log here: >> >> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console >> also >> >> copied below. >> >> >> >> I've tried so many things, including switching JVMs on master and >> agent, >> >> upgrading and switching to LTS, uninstalling and reinstalling a new >> agent >> >> service in a new working directory… Any tips or advice would be >> extremely >> >> appreciated; been trying to solve this all day and night since it >> started >> >> Thursday when the master rebooted during a job on the Windows agent. >> > >> > We have thins kind of errors on Windows when there are still running >> > processes that have a file descriptor opened. >> > You could log on to the machine and check if there are any "zombie" >> processes. >> > >> > Chris >> > >> >> >> >> Thanks in advance, >> >> Sean >> >> >> >> Full error: >> >> >> >> java.nio.file.AccessDeniedException: >> >> >> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only >> >> >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) >> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) >> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) >> >> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) >> >> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown >> Source) >> >> at java.nio.file.Files.deleteIfExists(Unknown Source) >> >> at hudson.Util.tryOnceDeleteFile(Util.java:297) >> >> at hudson.Util.deleteFile(Util.java:253) >> >> Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to >> >> JNLP4-connect connection from >> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 >> >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) >> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) >> >> at hudson.remoting.Channel.call(Channel.java:952) >> >> at hudson.FilePath.act(FilePath.java:998) >> >> at hudson.FilePath.act(FilePath.java:987) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1192) >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) >> >> >> at hudson.security.ACL.impersonate(ACL.java:290) >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) >> >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> Caused: java.io.IOException: Unable to delete >> >> >> 'C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only'. >> >> >> Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. >> >> at hudson.Util.deleteFile(Util.java:258) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1225) >> >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) >> >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) >> >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) >> >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) >> >> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1216) >> >> at hudson.FilePath.access$1100(FilePath.java:208) >> >> at hudson.FilePath$13.invoke(FilePath.java:1195) >> >> at hudson.FilePath$13.invoke(FilePath.java:1192) >> >> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:210) >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:53) >> >> at hudson.remoting.Request$2.run(Request.java:364) >> >> at >> >> >> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) >> >> >> at java.util.concurrent.FutureTask.run(Unknown Source) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> >> at hudson.remoting.Engine$1$1.run(Engine.java:94) >> >> at java.lang.Thread.run(Unknown Source) >> >> Also: java.nio.file.AccessDeniedException: >> >> >> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\stanc_helper.stan >> >> >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) >> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) >> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) >> >> at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source) >> >> at java.nio.file.spi.FileSystemProvider.newOutputStream(Unknown >> Source) >> >> at java.nio.file.Files.newOutputStream(Unknown Source) >> >> at hudson.util.IOUtils.copy(IOUtils.java:42) >> >> at hudson.FilePath.readFromTar(FilePath.java:2363) >> >> Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to >> >> JNLP4-connect connection from >> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 >> >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) >> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) >> >> at hudson.remoting.Channel.call(Channel.java:952) >> >> at hudson.FilePath.act(FilePath.java:998) >> >> at hudson.FilePath.act(FilePath.java:987) >> >> at hudson.FilePath.untar(FilePath.java:540) >> >> at >> >> >> org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) >> >> >> at hudson.security.ACL.impersonate(ACL.java:290) >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) >> >> >> at >> >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> Caused: java.io.IOException: Failed to extract StanSetup.tar.gz >> >> at hudson.FilePath.readFromTar(FilePath.java:2373) >> >> at hudson.FilePath.access$400(FilePath.java:208) >> >> at hudson.FilePath$4.invoke(FilePath.java:542) >> >> at hudson.FilePath$4.invoke(FilePath.java:540) >> >> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816) >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:210) >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:53) >> >> at hudson.remoting.Request$2.run(Request.java:364) >> >> at >> >> >> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) >> >> >> at java.util.concurrent.FutureTask.run(Unknown Source) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> >> at hudson.remoting.Engine$1$1.run(Engine.java:94) >> >> at java.lang.Thread.run(Unknown Source) >> >> Caused: java.io.IOException: remote file operation failed: >> >> >> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA >> >> >> at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from >> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 >> >> at hudson.FilePath.act(FilePath.java:1005) >> >> at hudson.FilePath.act(FilePath.java:987) >> >> at hudson.FilePath.untar(FilePath.java:540) >> >> at >> >> >> org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:129) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:74) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:61) >> >> >> Caused: java.io.IOException: remote file operation failed: >> >> >> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA >> >> >> at hudson.remoting.Channel@47c318f0:JNLP4-connect connection from >> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 >> >> at hudson.FilePath.act(FilePath.java:1005) >> >> at hudson.FilePath.act(FilePath.java:987) >> >> at hudson.FilePath.deleteRecursive(FilePath.java:1192) >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69) >> >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49) >> >> >> at hudson.security.ACL.impersonate(ACL.java:290) >> >> at >> >> >> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46) >> >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "Jenkins Users" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an >> >> email to [email protected]. >> >> To view this discussion on the web visit >> >> >> https://groups.google.com/d/msgid/jenkinsci-users/6eaa7b8d-d69b-4c5e-a818-2a94643ce96d%40googlegroups.com. >> >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Jenkins Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTjt-yWgMg58S%3DQNV7wy9r84J4hr270rjqTU-ck0sps%2BpA%40mail.gmail.com. >> >> > For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/3fce1831-0a1f-4a66-82d1-6180b5718b98%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/3fce1831-0a1f-4a66-82d1-6180b5718b98%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtEkX-BZFcB4_B8H-9vwjKZ740ET28r4jShavTrVjy5LCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
