Hi all, We ended up working around this problem by using a script to perform the clean up of the workspace. The script takes about 10 seconds to run, as opposed to the estimated hour it used to take the Java svn implementation before it started taking literally forever.
While we were at it, we decided to move to using a script to perform a platform specific, sparse directory checkout. This is the same script that our developers use to get only the stuff that they need to build on a given platform. No need to have pre-built Windows libraries checked out on a Mac, for example. In total, moving away from letting Jenkins and the underlying SVN library do the clean up and checkout has shaved 2 hours off of our build time. What we've lost is the ability to nicely associate svn changes with a given build on Jenkins. To attempt address this, I tried to have the build job that's downstream of the cleanup/checkout job do an SVN update of the workspace, but after watching it run for an hour, I gave up and terminated it. The same update of a sparse check out workspace with the command line svn tool takes 7 seconds. Honestly, I don't know if the current underlying Java svn lib simply cannot handle updating a sparse directory (which would be a bug), or it's just so slow that it's unreasonable to use with our source tree. Either way it looks like we're going to stick with the pragmatic approach of going "native" for portions of our jobs when Jenkins doesn't give us with the results we need. Best, -- Allen Cronce On Jun 21, 2012, at 7:10 AM, Allen Cronce wrote: > Hi all, > > One of our nightly builds has started regularly failing due to a hang. It > gets stuck at the very beginning of the job attempting to clean up the > workspace. When look at the console output, it's just got the spinning > progress wheel. > > The job in question is configured with the "Always check out a fresh copy" > SVN checkout strategy. The job runs on a build slave via ssh. The job > configuration itself has not changed in a long time. But the SVN project is > very large and very active. The only recent Jenkins environment changes have > been the addition of some plugins used by other jobs, including the "Jenkins > Workspace Cleanup Plugin" ironically. > > There is nothing interesting in the log when the failure happens. I only see > this when I manually stop the build after it's been trying to clean up for 6 > hours: > > --snip-- > java.lang.InterruptedException > at java.lang.Object.wait(Native Method) > at hudson.remoting.Request.call(Request.java:127) > at hudson.remoting.Channel.call(Channel.java:681) > at hudson.FilePath.act(FilePath.java:777) > at hudson.FilePath.act(FilePath.java:770) > at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:743) > at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:685) > at hudson.model.AbstractProject.checkout(AbstractProject.java:1197) > at > hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:579) > at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:468) > at hudson.model.Run.run(Run.java:1410) > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) > at hudson.model.ResourceController.execute(ResourceController.java:88) > at hudson.model.Executor.run(Executor.java:238) > --snip-- > > Note that operating under the assumption that maybe an open file is hanging > the build, I tried restarting the build slave, the restarting the build. Same > hang. The only work around has been to manually throw the workspace away on > the slave, then start the build. > > The problem sounds exactly like a post to this group with the subject > "jenkins hanging on build" from August 10, 2011. But I don't see any reply or > resolution. > > This problem is really killing us because we're in total crunch mode right > now and this is the first build in a series that takes nearly 6 hours. Every > time the build hangs, we have to restart it manually, which loses valuable > testing time. > > As a desperate work around attempt, I'm trying the Jenkins Workspace Cleanup > Plugin, which we recently installed, but is not being currently used with > this job. It's been churning away for 20 minutes with no visible progress at > the console level. But at least it seems to be doing something under the > hood. I see the java task on the client side as 100+% (it's a multi core > system), and according to fs_usage (it's a Mac) it is actually unlinking > files. Maybe someday it will finish... > > If anyone else has any experience with this problem or suggestions for a work > around, I'd appreciate it. > > Best, > -- > Allen Cronce
