Hi all
just thought of something : It seems that we are focussing mainly on interval triggers, (which is ok) suppose there is a source control error with a schedule trigger (nightly build), what would the wanted behaviour be? this has mainly to do how the retry will be implemented : ° wait 5 seconds and retry or ° place project back in the Q and wait for next integration with the interval trigger, both approaches are more or less the same I prefer to place it back in the Q (basically it means ignore source control exceptions until MaxSourceControlExceptions is reached) this means that the waiting period is longer (trigger interval), 30 seconds * 5 = 2.5 minutes that source control has to recuperate if you have wait 5 seconds before retry, and it means source control has 5 * 5 = 25 seconds to recuperate but with a schedule trigger, the next integration round may be : tomorrow any thoughts on this? or should we add MaxSourceControlExceptions on the trigger level ? (and when not set, take it from the project level) ( just brainstorming here ... ) with kind regards Ruben Willems On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems <[email protected]>wrote: > Hi > > That is even better, automatic recovery ! > +1 for me > > with kind regards > Ruben Willems > > On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> wrote: > >> I also really like the idea of having 1 exception build. >> >> But with Craigs proposed solution I'm not sure I see the point of stopping >> the project. >> If the network/source control system comes up again it feels wasted that >> the project has already been stopped. >> >> I think I would be happier with the following scenario. >> - if get modifications fails, it only logs the error (similar to >> pre-1.4.3). >> - it counts the number of successive failures >> - if the number of successive failures EQUALS the value in >> maxSourceControlRetries then it will log an error, change the build status >> to exception and CONTINUE RUNNING the project. >> - if the above state occurs, it will also run the publishers to send out >> any e-mails, notifications, etc. >> >> What happens when the project continue running after the failed build due >> to maxSourceControlRetries? >> - if get modifications fails, it continue forever to log the error >> (similar to pre-1.4.3) >> - it counts the number of successive failures up to Int32.MaxValue >> - it should NOT reach maxSourceControlRetries again and not trigger any >> new publishers. >> - as soon as the get modifications succeeds and a "real build" is started >> the exception count is reset again >> >> Comments? >> >> //P-J >> >> 2009/3/26 Ruben Willems <[email protected]> >> >> Hi Craig >>> >>> Sounds ok to me >>> >>> only a question : >>> when is it retried ? >>> immediately, or is the project placed back in the Q? >>> >>> if it is immediately, what would be the sleep/wait time before the retry >>> occurs? >>> >>> I like the idea of having 1 exception build, even if the >>> maxSourceControlRetry value is set to 5 >>> that is indeed a very great improvement over the current approach, where >>> every attempt would run the publisher block. >>> >>> >>> with kind regards >>> Ruben Willems >>> >>> >>> >>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < >>> [email protected]> wrote: >>> >>>> >>>> Hello all, >>>> >>>> I've been looking into this issue and seeing how we can improve >>>> things, since it obviously is a contentious issue. >>>> >>>> First, what we were trying to achieve: previously errors with source >>>> control were ignored or they would just crash the server. Obviously >>>> this state was not particularly good. The changes now include source >>>> control operations errors as build exceptions. Unfortunately, this now >>>> appears to have gone the other way and made CC.Net too sensitive to >>>> issues with source control. >>>> >>>> So what I am planning on doing is adding some tolerance to the >>>> process. There is already a project setting called >>>> maxSourceControlRetries, which currently sets the allowed number of >>>> source control failures before stopping the project. >>>> >>>> What I am going to do is modify the source control checking, so if get >>>> modifications fails, it only logs the error (similar to pre-1.4.3). >>>> However, it will count the number of failures - if the number of >>>> failures reaches the value in maxSourceControlRetries (by default this >>>> is five), then it will log an error, change the build status to >>>> exception and stop the project. If this state occurs, it will also run >>>> the publishers to send out any e-mails, notifications, etc. >>>> >>>> This way people now have some control over the sensitivity of source >>>> control checks. If you want to completely ignore this feature, set >>>> maxSourceControlRetries to some really high number (it's an Int32 >>>> though, so not too high), or if your source control/network/etc. is >>>> really, really good and you don't except any problems, set it to a low >>>> number (e.g. 0). >>>> >>>> What do people think about this approach? >>>> >>>> >>>> Craig >>>> >>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> wrote: >>>> > >>>> > >>> I now have 7 projects in "Exception" status (which are actually >>>> > >>> running) with failed builds and none of them have actually failed, >>>> > >>> they only failed because our server wasn't able to query svn >>>> because >>>> > >>> another build appears to have saturated available resources on the >>>> > >>> server.. >>>> > >>>> > >> ...and that's where our opinions differ: I see the situation as an >>>> actual >>>> > >> failure - if you aren't able to guarantee a successful build in a >>>> timely >>>> > >> fashion, it's a failure (along the lines of a typical "enterprise" >>>> SLA)... >>>> > >> just a question of filtering different failure notifications to >>>> different >>>> > >> people based on the cause. But then my opinion is just one of many, >>>> and >>>> > >> everyone is entitled to their own ;-) >>>> > >>>> > > Of course, and choice is good! >>>> > >>>> > > Anyway, I took Ruben's advice about setting up a single queue for >>>> all >>>> > > our projects, and that should resolve the problem. >>>> > >>>> > > I'm still not completely convinced that the underlying cause (svn >>>> log >>>> > > timeout) was due to the server or network, it may well be, but if >>>> more >>>> > > reports start coming in then there might be some other issue at >>>> work. >>>> > >>>> > I agree with Si. I would like the svn timeouts to be logged just as >>>> > they were pre-1.4.3, but I don't think the builds should enter the >>>> > failed state due to a 'checking for modifications' action. An >>>> > administrative notification can be sent, but that should only notify >>>> > the administrator of the build system, and not the 100 developers who >>>> > only need to focus on code modifications. >>>> > >>>> > Thanks, >>>> > Jon >>>> >>> >>> >> >
