Hi all
Another thing to think of : I placed the project into stopped because of the following reason : we once had some problems with a project on CCNetLive, there was a problem with the setup but the project kept polling the source control (sourceforge) After a time, the CCNet live server was blacklisted, so we could not even get the code of CCNet run. It took us quite some time to get us get the CCNet live server un-banned. and to prevent stuf like this in the future, I added the action that the project should be stopped when reaching the max amount of source control exceptions. This made perfect sense with the 1.4.3 release idea : when there is a source control problem, report it. Ok now it seems that this was a bit overdone, and during the discussion, we found better alternatives, but this i overlooked : The ability to stop the project so it will not be banned. Now I think there will not be many people needing this feature, but CCNet will. Now the question remaining is : where to configure this? at the project level, or at the server level? I will put the default to not stop the project once the amount has been reached. CCnet live can set the value to true so it will not get banned again. with kind regards Ruben Willems On Thu, Mar 26, 2009 at 10:43 AM, Ruben Willems <[email protected]>wrote: > Hi all > > > just thought of something : > It seems that we are focussing mainly on interval triggers, (which is ok) > > suppose there is a source control error with a schedule trigger (nightly > build), > what would the wanted behaviour be? > > this has mainly to do how the retry will be implemented : > ° wait 5 seconds and retry > or > ° place project back in the Q and wait for next integration > > > > with the interval trigger, both approaches are more or less the same > I prefer to place it back in the Q (basically it means ignore source > control exceptions until MaxSourceControlExceptions is reached) > this means that the waiting period is longer (trigger interval), 30 seconds > * 5 = 2.5 minutes that source control has to recuperate > > if you have wait 5 seconds before retry, and it means source control has 5 > * 5 = 25 seconds to recuperate > > > but with a schedule trigger, the next integration round may be : tomorrow > any thoughts on this? > > > or should we add MaxSourceControlExceptions on the trigger level ? (and > when not set, take it from the project level) > ( just brainstorming here ... ) > > > with kind regards > Ruben Willems > > > > On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems <[email protected]>wrote: > >> Hi >> >> That is even better, automatic recovery ! >> +1 for me >> >> with kind regards >> Ruben Willems >> >> On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> wrote: >> >>> I also really like the idea of having 1 exception build. >>> >>> But with Craigs proposed solution I'm not sure I see the point of >>> stopping the project. >>> If the network/source control system comes up again it feels wasted that >>> the project has already been stopped. >>> >>> I think I would be happier with the following scenario. >>> - if get modifications fails, it only logs the error (similar to >>> pre-1.4.3). >>> - it counts the number of successive failures >>> - if the number of successive failures EQUALS the value in >>> maxSourceControlRetries then it will log an error, change the build status >>> to exception and CONTINUE RUNNING the project. >>> - if the above state occurs, it will also run the publishers to send out >>> any e-mails, notifications, etc. >>> >>> What happens when the project continue running after the failed build due >>> to maxSourceControlRetries? >>> - if get modifications fails, it continue forever to log the error >>> (similar to pre-1.4.3) >>> - it counts the number of successive failures up to Int32.MaxValue >>> - it should NOT reach maxSourceControlRetries again and not trigger any >>> new publishers. >>> - as soon as the get modifications succeeds and a "real build" is started >>> the exception count is reset again >>> >>> Comments? >>> >>> //P-J >>> >>> 2009/3/26 Ruben Willems <[email protected]> >>> >>> Hi Craig >>>> >>>> Sounds ok to me >>>> >>>> only a question : >>>> when is it retried ? >>>> immediately, or is the project placed back in the Q? >>>> >>>> if it is immediately, what would be the sleep/wait time before the retry >>>> occurs? >>>> >>>> I like the idea of having 1 exception build, even if the >>>> maxSourceControlRetry value is set to 5 >>>> that is indeed a very great improvement over the current approach, where >>>> every attempt would run the publisher block. >>>> >>>> >>>> with kind regards >>>> Ruben Willems >>>> >>>> >>>> >>>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland < >>>> [email protected]> wrote: >>>> >>>>> >>>>> Hello all, >>>>> >>>>> I've been looking into this issue and seeing how we can improve >>>>> things, since it obviously is a contentious issue. >>>>> >>>>> First, what we were trying to achieve: previously errors with source >>>>> control were ignored or they would just crash the server. Obviously >>>>> this state was not particularly good. The changes now include source >>>>> control operations errors as build exceptions. Unfortunately, this now >>>>> appears to have gone the other way and made CC.Net too sensitive to >>>>> issues with source control. >>>>> >>>>> So what I am planning on doing is adding some tolerance to the >>>>> process. There is already a project setting called >>>>> maxSourceControlRetries, which currently sets the allowed number of >>>>> source control failures before stopping the project. >>>>> >>>>> What I am going to do is modify the source control checking, so if get >>>>> modifications fails, it only logs the error (similar to pre-1.4.3). >>>>> However, it will count the number of failures - if the number of >>>>> failures reaches the value in maxSourceControlRetries (by default this >>>>> is five), then it will log an error, change the build status to >>>>> exception and stop the project. If this state occurs, it will also run >>>>> the publishers to send out any e-mails, notifications, etc. >>>>> >>>>> This way people now have some control over the sensitivity of source >>>>> control checks. If you want to completely ignore this feature, set >>>>> maxSourceControlRetries to some really high number (it's an Int32 >>>>> though, so not too high), or if your source control/network/etc. is >>>>> really, really good and you don't except any problems, set it to a low >>>>> number (e.g. 0). >>>>> >>>>> What do people think about this approach? >>>>> >>>>> >>>>> Craig >>>>> >>>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote: >>>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> wrote: >>>>> > >>>>> > >>> I now have 7 projects in "Exception" status (which are actually >>>>> > >>> running) with failed builds and none of them have actually >>>>> failed, >>>>> > >>> they only failed because our server wasn't able to query svn >>>>> because >>>>> > >>> another build appears to have saturated available resources on >>>>> the >>>>> > >>> server.. >>>>> > >>>>> > >> ...and that's where our opinions differ: I see the situation as an >>>>> actual >>>>> > >> failure - if you aren't able to guarantee a successful build in a >>>>> timely >>>>> > >> fashion, it's a failure (along the lines of a typical "enterprise" >>>>> SLA)... >>>>> > >> just a question of filtering different failure notifications to >>>>> different >>>>> > >> people based on the cause. But then my opinion is just one of >>>>> many, and >>>>> > >> everyone is entitled to their own ;-) >>>>> > >>>>> > > Of course, and choice is good! >>>>> > >>>>> > > Anyway, I took Ruben's advice about setting up a single queue for >>>>> all >>>>> > > our projects, and that should resolve the problem. >>>>> > >>>>> > > I'm still not completely convinced that the underlying cause (svn >>>>> log >>>>> > > timeout) was due to the server or network, it may well be, but if >>>>> more >>>>> > > reports start coming in then there might be some other issue at >>>>> work. >>>>> > >>>>> > I agree with Si. I would like the svn timeouts to be logged just as >>>>> > they were pre-1.4.3, but I don't think the builds should enter the >>>>> > failed state due to a 'checking for modifications' action. An >>>>> > administrative notification can be sent, but that should only notify >>>>> > the administrator of the build system, and not the 100 developers who >>>>> > only need to focus on code modifications. >>>>> > >>>>> > Thanks, >>>>> > Jon >>>>> >>>> >>>> >>> >> >
