I also really like the idea of having 1 exception build. But with Craigs proposed solution I'm not sure I see the point of stopping the project. If the network/source control system comes up again it feels wasted that the project has already been stopped.
I think I would be happier with the following scenario. - if get modifications fails, it only logs the error (similar to pre-1.4.3). - it counts the number of successive failures - if the number of successive failures EQUALS the value in maxSourceControlRetries then it will log an error, change the build status to exception and CONTINUE RUNNING the project. - if the above state occurs, it will also run the publishers to send out any e-mails, notifications, etc. What happens when the project continue running after the failed build due to maxSourceControlRetries? - if get modifications fails, it continue forever to log the error (similar to pre-1.4.3) - it counts the number of successive failures up to Int32.MaxValue - it should NOT reach maxSourceControlRetries again and not trigger any new publishers. - as soon as the get modifications succeeds and a "real build" is started the exception count is reset again Comments? //P-J 2009/3/26 Ruben Willems <[email protected]> > Hi Craig > > Sounds ok to me > > only a question : > when is it retried ? > immediately, or is the project placed back in the Q? > > if it is immediately, what would be the sleep/wait time before the retry > occurs? > > I like the idea of having 1 exception build, even if the > maxSourceControlRetry value is set to 5 > that is indeed a very great improvement over the current approach, where > every attempt would run the publisher block. > > > with kind regards > Ruben Willems > > > > On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < > [email protected]> wrote: > >> >> Hello all, >> >> I've been looking into this issue and seeing how we can improve >> things, since it obviously is a contentious issue. >> >> First, what we were trying to achieve: previously errors with source >> control were ignored or they would just crash the server. Obviously >> this state was not particularly good. The changes now include source >> control operations errors as build exceptions. Unfortunately, this now >> appears to have gone the other way and made CC.Net too sensitive to >> issues with source control. >> >> So what I am planning on doing is adding some tolerance to the >> process. There is already a project setting called >> maxSourceControlRetries, which currently sets the allowed number of >> source control failures before stopping the project. >> >> What I am going to do is modify the source control checking, so if get >> modifications fails, it only logs the error (similar to pre-1.4.3). >> However, it will count the number of failures - if the number of >> failures reaches the value in maxSourceControlRetries (by default this >> is five), then it will log an error, change the build status to >> exception and stop the project. If this state occurs, it will also run >> the publishers to send out any e-mails, notifications, etc. >> >> This way people now have some control over the sensitivity of source >> control checks. If you want to completely ignore this feature, set >> maxSourceControlRetries to some really high number (it's an Int32 >> though, so not too high), or if your source control/network/etc. is >> really, really good and you don't except any problems, set it to a low >> number (e.g. 0). >> >> What do people think about this approach? >> >> >> Craig >> >> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> wrote: >> > >> > >>> I now have 7 projects in "Exception" status (which are actually >> > >>> running) with failed builds and none of them have actually failed, >> > >>> they only failed because our server wasn't able to query svn because >> > >>> another build appears to have saturated available resources on the >> > >>> server.. >> > >> > >> ...and that's where our opinions differ: I see the situation as an >> actual >> > >> failure - if you aren't able to guarantee a successful build in a >> timely >> > >> fashion, it's a failure (along the lines of a typical "enterprise" >> SLA)... >> > >> just a question of filtering different failure notifications to >> different >> > >> people based on the cause. But then my opinion is just one of many, >> and >> > >> everyone is entitled to their own ;-) >> > >> > > Of course, and choice is good! >> > >> > > Anyway, I took Ruben's advice about setting up a single queue for all >> > > our projects, and that should resolve the problem. >> > >> > > I'm still not completely convinced that the underlying cause (svn log >> > > timeout) was due to the server or network, it may well be, but if more >> > > reports start coming in then there might be some other issue at work. >> > >> > I agree with Si. I would like the svn timeouts to be logged just as >> > they were pre-1.4.3, but I don't think the builds should enter the >> > failed state due to a 'checking for modifications' action. An >> > administrative notification can be sent, but that should only notify >> > the administrator of the build system, and not the 100 developers who >> > only need to focus on code modifications. >> > >> > Thanks, >> > Jon >> > >
