Hi no, Craig did a change he mentioned before, this does not inlcude my proposed changes yet I hope to do that one of these days.
with kind regards Ruben Willems On Tue, Mar 31, 2009 at 8:31 AM, Per-Jonny Käck <[email protected]> wrote: > Does this mean that everything is in for the 1.4.4 release now? > Unless someone finds any issues with these sticky exception changes? > > //P-J > > 2009/3/31 Craig Sutherland <[email protected]> > > >> Hello all, >> >> I've made the basic changes that I described earlier - by default a >> failure during the get modifications stage will return the build to >> it's previous state and not run any publishers (i.e. similar >> functionality to pre-1.4.3). Once the maximum number of retries is >> hit, it will change the status of the build to Exception, run the >> publishers and stop the project (i.e the 1.4.3 functionality). >> >> We'll look at refining this functionality over time, but hopefully >> this will provide the basics for most people. These changes can be >> downloaded from http://ccnetlive.thoughtworks.com/CCNet-builds/1.4.3/ >> (choose one of the latest builds). >> >> Let us know if you find any issues, >> >> >> Craig >> >> On Mar 26, 9:43 pm, Ruben Willems <[email protected]> wrote: >> > Hi all >> > >> > just thought of something : >> > It seems that we are focussing mainly on interval triggers, (which is >> ok) >> > >> > suppose there is a source control error with a schedule trigger (nightly >> > build), >> > what would the wanted behaviour be? >> > >> > this has mainly to do how the retry will be implemented : >> > ° wait 5 seconds and retry >> > or >> > ° place project back in the Q and wait for next integration >> > >> > with the interval trigger, both approaches are more or less the same >> > I prefer to place it back in the Q (basically it means ignore source >> control >> > exceptions until MaxSourceControlExceptions is reached) >> > this means that the waiting period is longer (trigger interval), 30 >> seconds >> > * 5 = 2.5 minutes that source control has to recuperate >> > >> > if you have wait 5 seconds before retry, and it means source control has >> 5 * >> > 5 = 25 seconds to recuperate >> > >> > but with a schedule trigger, the next integration round may be : >> tomorrow >> > any thoughts on this? >> > >> > or should we add MaxSourceControlExceptions on the trigger level ? (and >> when >> > not set, take it from the project level) >> > ( just brainstorming here ... ) >> > >> > with kind regards >> > Ruben Willems >> > >> > On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems <[email protected] >> >wrote: >> > >> > > Hi >> > >> > > That is even better, automatic recovery ! >> > > +1 for me >> > >> > > with kind regards >> > > Ruben Willems >> > >> > > On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> >> wrote: >> > >> > >> I also really like the idea of having 1 exception build. >> > >> > >> But with Craigs proposed solution I'm not sure I see the point of >> stopping >> > >> the project. >> > >> If the network/source control system comes up again it feels wasted >> that >> > >> the project has already been stopped. >> > >> > >> I think I would be happier with the following scenario. >> > >> - if get modifications fails, it only logs the error (similar to >> > >> pre-1.4.3). >> > >> - it counts the number of successive failures >> > >> - if the number of successive failures EQUALS the value in >> > >> maxSourceControlRetries then it will log an error, change the build >> status >> > >> to exception and CONTINUE RUNNING the project. >> > >> - if the above state occurs, it will also run the publishers to send >> out >> > >> any e-mails, notifications, etc. >> > >> > >> What happens when the project continue running after the failed build >> due >> > >> to maxSourceControlRetries? >> > >> - if get modifications fails, it continue forever to log the error >> > >> (similar to pre-1.4.3) >> > >> - it counts the number of successive failures up to Int32.MaxValue >> > >> - it should NOT reach maxSourceControlRetries again and not trigger >> any >> > >> new publishers. >> > >> - as soon as the get modifications succeeds and a "real build" is >> started >> > >> the exception count is reset again >> > >> > >> Comments? >> > >> > >> //P-J >> > >> > >> 2009/3/26 Ruben Willems <[email protected]> >> > >> > >> Hi Craig >> > >> > >>> Sounds ok to me >> > >> > >>> only a question : >> > >>> when is it retried ? >> > >>> immediately, or is the project placed back in the Q? >> > >> > >>> if it is immediately, what would be the sleep/wait time before the >> retry >> > >>> occurs? >> > >> > >>> I like the idea of having 1 exception build, even if the >> > >>> maxSourceControlRetry value is set to 5 >> > >>> that is indeed a very great improvement over the current approach, >> where >> > >>> every attempt would run the publisher block. >> > >> > >>> with kind regards >> > >>> Ruben Willems >> > >> > >>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < >> > >>> [email protected]> wrote: >> > >> > >>>> Hello all, >> > >> > >>>> I've been looking into this issue and seeing how we can improve >> > >>>> things, since it obviously is a contentious issue. >> > >> > >>>> First, what we were trying to achieve: previously errors with >> source >> > >>>> control were ignored or they would just crash the server. Obviously >> > >>>> this state was not particularly good. The changes now include >> source >> > >>>> control operations errors as build exceptions. Unfortunately, this >> now >> > >>>> appears to have gone the other way and made CC.Net too sensitive to >> > >>>> issues with source control. >> > >> > >>>> So what I am planning on doing is adding some tolerance to the >> > >>>> process. There is already a project setting called >> > >>>> maxSourceControlRetries, which currently sets the allowed number of >> > >>>> source control failures before stopping the project. >> > >> > >>>> What I am going to do is modify the source control checking, so if >> get >> > >>>> modifications fails, it only logs the error (similar to pre-1.4.3). >> > >>>> However, it will count the number of failures - if the number of >> > >>>> failures reaches the value in maxSourceControlRetries (by default >> this >> > >>>> is five), then it will log an error, change the build status to >> > >>>> exception and stop the project. If this state occurs, it will also >> run >> > >>>> the publishers to send out any e-mails, notifications, etc. >> > >> > >>>> This way people now have some control over the sensitivity of >> source >> > >>>> control checks. If you want to completely ignore this feature, set >> > >>>> maxSourceControlRetries to some really high number (it's an Int32 >> > >>>> though, so not too high), or if your source control/network/etc. is >> > >>>> really, really good and you don't except any problems, set it to a >> low >> > >>>> number (e.g. 0). >> > >> > >>>> What do people think about this approach? >> > >> > >>>> Craig >> > >> > >>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >> > >>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> wrote: >> > >> > >>>> > >>> I now have 7 projects in "Exception" status (which are >> actually >> > >>>> > >>> running) with failed builds and none of them have actually >> failed, >> > >>>> > >>> they only failed because our server wasn't able to query svn >> > >>>> because >> > >>>> > >>> another build appears to have saturated available resources >> on the >> > >>>> > >>> server.. >> > >> > >>>> > >> ...and that's where our opinions differ: I see the situation >> as an >> > >>>> actual >> > >>>> > >> failure - if you aren't able to guarantee a successful build >> in a >> > >>>> timely >> > >>>> > >> fashion, it's a failure (along the lines of a typical >> "enterprise" >> > >>>> SLA)... >> > >>>> > >> just a question of filtering different failure notifications >> to >> > >>>> different >> > >>>> > >> people based on the cause. But then my opinion is just one of >> many, >> > >>>> and >> > >>>> > >> everyone is entitled to their own ;-) >> > >> > >>>> > > Of course, and choice is good! >> > >> > >>>> > > Anyway, I took Ruben's advice about setting up a single queue >> for >> > >>>> all >> > >>>> > > our projects, and that should resolve the problem. >> > >> > >>>> > > I'm still not completely convinced that the underlying cause >> (svn >> > >>>> log >> > >>>> > > timeout) was due to the server or network, it may well be, but >> if >> > >>>> more >> > >>>> > > reports start coming in then there might be some other issue at >> > >>>> work. >> > >> > >>>> > I agree with Si. I would like the svn timeouts to be logged just >> as >> > >>>> > they were pre-1.4.3, but I don't think the builds should enter >> the >> > >>>> > failed state due to a 'checking for modifications' action. An >> > >>>> > administrative notification can be sent, but that should only >> notify >> > >>>> > the administrator of the build system, and not the 100 developers >> who >> > >>>> > only need to focus on code modifications. >> > >> > >>>> > Thanks, >> > >>>> > Jon >> > >
