Hi
it is comitted to the trunk, but ccnet live has a problem which I can not reproduce locally it should work, so if someone will try it out with kind regards Ruben Willems On Tue, Mar 31, 2009 at 10:49 AM, Ruben Willems <[email protected]>wrote: > Hi all > > I created an issue in Jira > http://jira.public.thoughtworks.org/browse/CCNET-1441 > > working on it now ... > > > with kind regards > Ruben Willems > > > On Tue, Mar 31, 2009 at 8:37 AM, Ruben Willems <[email protected]>wrote: > >> Hi >> >> no, Craig did a change he mentioned before, this does not inlcude my >> proposed changes yet >> I hope to do that one of these days. >> >> >> with kind regards >> Ruben Willems >> >> On Tue, Mar 31, 2009 at 8:31 AM, Per-Jonny Käck <[email protected]> wrote: >> >>> Does this mean that everything is in for the 1.4.4 release now? >>> Unless someone finds any issues with these sticky exception changes? >>> >>> //P-J >>> >>> 2009/3/31 Craig Sutherland <[email protected]> >>> >>> >>>> Hello all, >>>> >>>> I've made the basic changes that I described earlier - by default a >>>> failure during the get modifications stage will return the build to >>>> it's previous state and not run any publishers (i.e. similar >>>> functionality to pre-1.4.3). Once the maximum number of retries is >>>> hit, it will change the status of the build to Exception, run the >>>> publishers and stop the project (i.e the 1.4.3 functionality). >>>> >>>> We'll look at refining this functionality over time, but hopefully >>>> this will provide the basics for most people. These changes can be >>>> downloaded from http://ccnetlive.thoughtworks.com/CCNet-builds/1.4.3/ >>>> (choose one of the latest builds). >>>> >>>> Let us know if you find any issues, >>>> >>>> >>>> Craig >>>> >>>> On Mar 26, 9:43 pm, Ruben Willems <[email protected]> wrote: >>>> > Hi all >>>> > >>>> > just thought of something : >>>> > It seems that we are focussing mainly on interval triggers, (which is >>>> ok) >>>> > >>>> > suppose there is a source control error with a schedule trigger >>>> (nightly >>>> > build), >>>> > what would the wanted behaviour be? >>>> > >>>> > this has mainly to do how the retry will be implemented : >>>> > ° wait 5 seconds and retry >>>> > or >>>> > ° place project back in the Q and wait for next integration >>>> > >>>> > with the interval trigger, both approaches are more or less the same >>>> > I prefer to place it back in the Q (basically it means ignore source >>>> control >>>> > exceptions until MaxSourceControlExceptions is reached) >>>> > this means that the waiting period is longer (trigger interval), 30 >>>> seconds >>>> > * 5 = 2.5 minutes that source control has to recuperate >>>> > >>>> > if you have wait 5 seconds before retry, and it means source control >>>> has 5 * >>>> > 5 = 25 seconds to recuperate >>>> > >>>> > but with a schedule trigger, the next integration round may be : >>>> tomorrow >>>> > any thoughts on this? >>>> > >>>> > or should we add MaxSourceControlExceptions on the trigger level ? >>>> (and when >>>> > not set, take it from the project level) >>>> > ( just brainstorming here ... ) >>>> > >>>> > with kind regards >>>> > Ruben Willems >>>> > >>>> > On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems < >>>> [email protected]>wrote: >>>> > >>>> > > Hi >>>> > >>>> > > That is even better, automatic recovery ! >>>> > > +1 for me >>>> > >>>> > > with kind regards >>>> > > Ruben Willems >>>> > >>>> > > On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> >>>> wrote: >>>> > >>>> > >> I also really like the idea of having 1 exception build. >>>> > >>>> > >> But with Craigs proposed solution I'm not sure I see the point of >>>> stopping >>>> > >> the project. >>>> > >> If the network/source control system comes up again it feels wasted >>>> that >>>> > >> the project has already been stopped. >>>> > >>>> > >> I think I would be happier with the following scenario. >>>> > >> - if get modifications fails, it only logs the error (similar to >>>> > >> pre-1.4.3). >>>> > >> - it counts the number of successive failures >>>> > >> - if the number of successive failures EQUALS the value in >>>> > >> maxSourceControlRetries then it will log an error, change the build >>>> status >>>> > >> to exception and CONTINUE RUNNING the project. >>>> > >> - if the above state occurs, it will also run the publishers to >>>> send out >>>> > >> any e-mails, notifications, etc. >>>> > >>>> > >> What happens when the project continue running after the failed >>>> build due >>>> > >> to maxSourceControlRetries? >>>> > >> - if get modifications fails, it continue forever to log the error >>>> > >> (similar to pre-1.4.3) >>>> > >> - it counts the number of successive failures up to Int32.MaxValue >>>> > >> - it should NOT reach maxSourceControlRetries again and not trigger >>>> any >>>> > >> new publishers. >>>> > >> - as soon as the get modifications succeeds and a "real build" is >>>> started >>>> > >> the exception count is reset again >>>> > >>>> > >> Comments? >>>> > >>>> > >> //P-J >>>> > >>>> > >> 2009/3/26 Ruben Willems <[email protected]> >>>> > >>>> > >> Hi Craig >>>> > >>>> > >>> Sounds ok to me >>>> > >>>> > >>> only a question : >>>> > >>> when is it retried ? >>>> > >>> immediately, or is the project placed back in the Q? >>>> > >>>> > >>> if it is immediately, what would be the sleep/wait time before the >>>> retry >>>> > >>> occurs? >>>> > >>>> > >>> I like the idea of having 1 exception build, even if the >>>> > >>> maxSourceControlRetry value is set to 5 >>>> > >>> that is indeed a very great improvement over the current approach, >>>> where >>>> > >>> every attempt would run the publisher block. >>>> > >>>> > >>> with kind regards >>>> > >>> Ruben Willems >>>> > >>>> > >>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < >>>> > >>> [email protected]> wrote: >>>> > >>>> > >>>> Hello all, >>>> > >>>> > >>>> I've been looking into this issue and seeing how we can improve >>>> > >>>> things, since it obviously is a contentious issue. >>>> > >>>> > >>>> First, what we were trying to achieve: previously errors with >>>> source >>>> > >>>> control were ignored or they would just crash the server. >>>> Obviously >>>> > >>>> this state was not particularly good. The changes now include >>>> source >>>> > >>>> control operations errors as build exceptions. Unfortunately, >>>> this now >>>> > >>>> appears to have gone the other way and made CC.Net too sensitive >>>> to >>>> > >>>> issues with source control. >>>> > >>>> > >>>> So what I am planning on doing is adding some tolerance to the >>>> > >>>> process. There is already a project setting called >>>> > >>>> maxSourceControlRetries, which currently sets the allowed number >>>> of >>>> > >>>> source control failures before stopping the project. >>>> > >>>> > >>>> What I am going to do is modify the source control checking, so >>>> if get >>>> > >>>> modifications fails, it only logs the error (similar to >>>> pre-1.4.3). >>>> > >>>> However, it will count the number of failures - if the number of >>>> > >>>> failures reaches the value in maxSourceControlRetries (by default >>>> this >>>> > >>>> is five), then it will log an error, change the build status to >>>> > >>>> exception and stop the project. If this state occurs, it will >>>> also run >>>> > >>>> the publishers to send out any e-mails, notifications, etc. >>>> > >>>> > >>>> This way people now have some control over the sensitivity of >>>> source >>>> > >>>> control checks. If you want to completely ignore this feature, >>>> set >>>> > >>>> maxSourceControlRetries to some really high number (it's an Int32 >>>> > >>>> though, so not too high), or if your source control/network/etc. >>>> is >>>> > >>>> really, really good and you don't except any problems, set it to >>>> a low >>>> > >>>> number (e.g. 0). >>>> > >>>> > >>>> What do people think about this approach? >>>> > >>>> > >>>> Craig >>>> > >>>> > >>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >>>> > >>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> >>>> wrote: >>>> > >>>> > >>>> > >>> I now have 7 projects in "Exception" status (which are >>>> actually >>>> > >>>> > >>> running) with failed builds and none of them have actually >>>> failed, >>>> > >>>> > >>> they only failed because our server wasn't able to query >>>> svn >>>> > >>>> because >>>> > >>>> > >>> another build appears to have saturated available resources >>>> on the >>>> > >>>> > >>> server.. >>>> > >>>> > >>>> > >> ...and that's where our opinions differ: I see the situation >>>> as an >>>> > >>>> actual >>>> > >>>> > >> failure - if you aren't able to guarantee a successful build >>>> in a >>>> > >>>> timely >>>> > >>>> > >> fashion, it's a failure (along the lines of a typical >>>> "enterprise" >>>> > >>>> SLA)... >>>> > >>>> > >> just a question of filtering different failure notifications >>>> to >>>> > >>>> different >>>> > >>>> > >> people based on the cause. But then my opinion is just one >>>> of many, >>>> > >>>> and >>>> > >>>> > >> everyone is entitled to their own ;-) >>>> > >>>> > >>>> > > Of course, and choice is good! >>>> > >>>> > >>>> > > Anyway, I took Ruben's advice about setting up a single queue >>>> for >>>> > >>>> all >>>> > >>>> > > our projects, and that should resolve the problem. >>>> > >>>> > >>>> > > I'm still not completely convinced that the underlying cause >>>> (svn >>>> > >>>> log >>>> > >>>> > > timeout) was due to the server or network, it may well be, >>>> but if >>>> > >>>> more >>>> > >>>> > > reports start coming in then there might be some other issue >>>> at >>>> > >>>> work. >>>> > >>>> > >>>> > I agree with Si. I would like the svn timeouts to be logged >>>> just as >>>> > >>>> > they were pre-1.4.3, but I don't think the builds should enter >>>> the >>>> > >>>> > failed state due to a 'checking for modifications' action. An >>>> > >>>> > administrative notification can be sent, but that should only >>>> notify >>>> > >>>> > the administrator of the build system, and not the 100 >>>> developers who >>>> > >>>> > only need to focus on code modifications. >>>> > >>>> > >>>> > Thanks, >>>> > >>>> > Jon >>>> >>> >>> >> >
