Hi all I created an issue in Jira http://jira.public.thoughtworks.org/browse/CCNET-1441
working on it now ... with kind regards Ruben Willems On Tue, Mar 31, 2009 at 8:37 AM, Ruben Willems <[email protected]>wrote: > Hi > > no, Craig did a change he mentioned before, this does not inlcude my > proposed changes yet > I hope to do that one of these days. > > > with kind regards > Ruben Willems > > On Tue, Mar 31, 2009 at 8:31 AM, Per-Jonny Käck <[email protected]> wrote: > >> Does this mean that everything is in for the 1.4.4 release now? >> Unless someone finds any issues with these sticky exception changes? >> >> //P-J >> >> 2009/3/31 Craig Sutherland <[email protected]> >> >> >>> Hello all, >>> >>> I've made the basic changes that I described earlier - by default a >>> failure during the get modifications stage will return the build to >>> it's previous state and not run any publishers (i.e. similar >>> functionality to pre-1.4.3). Once the maximum number of retries is >>> hit, it will change the status of the build to Exception, run the >>> publishers and stop the project (i.e the 1.4.3 functionality). >>> >>> We'll look at refining this functionality over time, but hopefully >>> this will provide the basics for most people. These changes can be >>> downloaded from http://ccnetlive.thoughtworks.com/CCNet-builds/1.4.3/ >>> (choose one of the latest builds). >>> >>> Let us know if you find any issues, >>> >>> >>> Craig >>> >>> On Mar 26, 9:43 pm, Ruben Willems <[email protected]> wrote: >>> > Hi all >>> > >>> > just thought of something : >>> > It seems that we are focussing mainly on interval triggers, (which is >>> ok) >>> > >>> > suppose there is a source control error with a schedule trigger >>> (nightly >>> > build), >>> > what would the wanted behaviour be? >>> > >>> > this has mainly to do how the retry will be implemented : >>> > ° wait 5 seconds and retry >>> > or >>> > ° place project back in the Q and wait for next integration >>> > >>> > with the interval trigger, both approaches are more or less the same >>> > I prefer to place it back in the Q (basically it means ignore source >>> control >>> > exceptions until MaxSourceControlExceptions is reached) >>> > this means that the waiting period is longer (trigger interval), 30 >>> seconds >>> > * 5 = 2.5 minutes that source control has to recuperate >>> > >>> > if you have wait 5 seconds before retry, and it means source control >>> has 5 * >>> > 5 = 25 seconds to recuperate >>> > >>> > but with a schedule trigger, the next integration round may be : >>> tomorrow >>> > any thoughts on this? >>> > >>> > or should we add MaxSourceControlExceptions on the trigger level ? (and >>> when >>> > not set, take it from the project level) >>> > ( just brainstorming here ... ) >>> > >>> > with kind regards >>> > Ruben Willems >>> > >>> > On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems < >>> [email protected]>wrote: >>> > >>> > > Hi >>> > >>> > > That is even better, automatic recovery ! >>> > > +1 for me >>> > >>> > > with kind regards >>> > > Ruben Willems >>> > >>> > > On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> >>> wrote: >>> > >>> > >> I also really like the idea of having 1 exception build. >>> > >>> > >> But with Craigs proposed solution I'm not sure I see the point of >>> stopping >>> > >> the project. >>> > >> If the network/source control system comes up again it feels wasted >>> that >>> > >> the project has already been stopped. >>> > >>> > >> I think I would be happier with the following scenario. >>> > >> - if get modifications fails, it only logs the error (similar to >>> > >> pre-1.4.3). >>> > >> - it counts the number of successive failures >>> > >> - if the number of successive failures EQUALS the value in >>> > >> maxSourceControlRetries then it will log an error, change the build >>> status >>> > >> to exception and CONTINUE RUNNING the project. >>> > >> - if the above state occurs, it will also run the publishers to send >>> out >>> > >> any e-mails, notifications, etc. >>> > >>> > >> What happens when the project continue running after the failed >>> build due >>> > >> to maxSourceControlRetries? >>> > >> - if get modifications fails, it continue forever to log the error >>> > >> (similar to pre-1.4.3) >>> > >> - it counts the number of successive failures up to Int32.MaxValue >>> > >> - it should NOT reach maxSourceControlRetries again and not trigger >>> any >>> > >> new publishers. >>> > >> - as soon as the get modifications succeeds and a "real build" is >>> started >>> > >> the exception count is reset again >>> > >>> > >> Comments? >>> > >>> > >> //P-J >>> > >>> > >> 2009/3/26 Ruben Willems <[email protected]> >>> > >>> > >> Hi Craig >>> > >>> > >>> Sounds ok to me >>> > >>> > >>> only a question : >>> > >>> when is it retried ? >>> > >>> immediately, or is the project placed back in the Q? >>> > >>> > >>> if it is immediately, what would be the sleep/wait time before the >>> retry >>> > >>> occurs? >>> > >>> > >>> I like the idea of having 1 exception build, even if the >>> > >>> maxSourceControlRetry value is set to 5 >>> > >>> that is indeed a very great improvement over the current approach, >>> where >>> > >>> every attempt would run the publisher block. >>> > >>> > >>> with kind regards >>> > >>> Ruben Willems >>> > >>> > >>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < >>> > >>> [email protected]> wrote: >>> > >>> > >>>> Hello all, >>> > >>> > >>>> I've been looking into this issue and seeing how we can improve >>> > >>>> things, since it obviously is a contentious issue. >>> > >>> > >>>> First, what we were trying to achieve: previously errors with >>> source >>> > >>>> control were ignored or they would just crash the server. >>> Obviously >>> > >>>> this state was not particularly good. The changes now include >>> source >>> > >>>> control operations errors as build exceptions. Unfortunately, this >>> now >>> > >>>> appears to have gone the other way and made CC.Net too sensitive >>> to >>> > >>>> issues with source control. >>> > >>> > >>>> So what I am planning on doing is adding some tolerance to the >>> > >>>> process. There is already a project setting called >>> > >>>> maxSourceControlRetries, which currently sets the allowed number >>> of >>> > >>>> source control failures before stopping the project. >>> > >>> > >>>> What I am going to do is modify the source control checking, so if >>> get >>> > >>>> modifications fails, it only logs the error (similar to >>> pre-1.4.3). >>> > >>>> However, it will count the number of failures - if the number of >>> > >>>> failures reaches the value in maxSourceControlRetries (by default >>> this >>> > >>>> is five), then it will log an error, change the build status to >>> > >>>> exception and stop the project. If this state occurs, it will also >>> run >>> > >>>> the publishers to send out any e-mails, notifications, etc. >>> > >>> > >>>> This way people now have some control over the sensitivity of >>> source >>> > >>>> control checks. If you want to completely ignore this feature, set >>> > >>>> maxSourceControlRetries to some really high number (it's an Int32 >>> > >>>> though, so not too high), or if your source control/network/etc. >>> is >>> > >>>> really, really good and you don't except any problems, set it to a >>> low >>> > >>>> number (e.g. 0). >>> > >>> > >>>> What do people think about this approach? >>> > >>> > >>>> Craig >>> > >>> > >>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >>> > >>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> >>> wrote: >>> > >>> > >>>> > >>> I now have 7 projects in "Exception" status (which are >>> actually >>> > >>>> > >>> running) with failed builds and none of them have actually >>> failed, >>> > >>>> > >>> they only failed because our server wasn't able to query svn >>> > >>>> because >>> > >>>> > >>> another build appears to have saturated available resources >>> on the >>> > >>>> > >>> server.. >>> > >>> > >>>> > >> ...and that's where our opinions differ: I see the situation >>> as an >>> > >>>> actual >>> > >>>> > >> failure - if you aren't able to guarantee a successful build >>> in a >>> > >>>> timely >>> > >>>> > >> fashion, it's a failure (along the lines of a typical >>> "enterprise" >>> > >>>> SLA)... >>> > >>>> > >> just a question of filtering different failure notifications >>> to >>> > >>>> different >>> > >>>> > >> people based on the cause. But then my opinion is just one of >>> many, >>> > >>>> and >>> > >>>> > >> everyone is entitled to their own ;-) >>> > >>> > >>>> > > Of course, and choice is good! >>> > >>> > >>>> > > Anyway, I took Ruben's advice about setting up a single queue >>> for >>> > >>>> all >>> > >>>> > > our projects, and that should resolve the problem. >>> > >>> > >>>> > > I'm still not completely convinced that the underlying cause >>> (svn >>> > >>>> log >>> > >>>> > > timeout) was due to the server or network, it may well be, but >>> if >>> > >>>> more >>> > >>>> > > reports start coming in then there might be some other issue >>> at >>> > >>>> work. >>> > >>> > >>>> > I agree with Si. I would like the svn timeouts to be logged >>> just as >>> > >>>> > they were pre-1.4.3, but I don't think the builds should enter >>> the >>> > >>>> > failed state due to a 'checking for modifications' action. An >>> > >>>> > administrative notification can be sent, but that should only >>> notify >>> > >>>> > the administrator of the build system, and not the 100 >>> developers who >>> > >>>> > only need to focus on code modifications. >>> > >>> > >>>> > Thanks, >>> > >>>> > Jon >>> >> >> >
