Hi
another trigger kind : force build only :
so a project with a <triggers/> section.
i think in this situation it is best that the build state is set to
exceptioned immediately.
the reason :
This trigger is used for manual intervention/launching specific actions.
now, suppose we only publish after the 5th source control error (the
default)
the following situation is possible :
it is time to place the new software into production:
QA manager presses the release button ...
nothing seems to happen
he pushes it again
nothing seems to happen
QA manager frownses, and pushes again
nothing seems to happen
QA manager getting pretty frustraded, hitting the poor mouse button again
nothing seems to happen
QA manager now really punished the poor plastic rodent
Error appears
QA manager : WHY can't this freaking system tell me this in the first place
==> how do we avoid this?
setting the maxSourceControl to 0 ?
but this means that the QA manager must first press the start button again
Setting the state to Exception the first time, immediately informed the poor
QA manager there was
a problem, and he would get in touch with the network guys/source control
guys to see where the problem was.
and when those guys gave the ok, he could again press the release button.
(project was not stopped yet)
It seems more and more that we need to explicitely to configure :
° when does a project reach the stopped state (if ever),
--> stop the project if maxSourceControlExceptions has reached or not
° report source control problems on every request or only after X attempts
--> only report when maxSourceControlExceptions has been reached, or on
every attempt
and a variation on this, when maxSourceControlExceptions has been
reached, set the counter back
to 0 or not. Setting it to 0 will report again the source source
control problem (if the project was not stopped)
if not setting to 0, the counter will increase forever until the
source control comes up again, or somebody notices
the problem and fixes it.
==> getting 1 warning or keep getting warnings untill fixed
We need at least 3 parameters I think :
a) maxSourceControlExceptions : integer, we have this
b) StopProjectOnReachingRetryAmount : boolean : stop the project or not
b) SourceControlErrorHandlingPolicy (what a name) which is an enum, with
options
ReportEveryFailure : sets the project state
to exception every integration, as in the 1.4.3 release
ReportOnRetryAmount : retries X amounts, before
setting project to exception state, only 1 time reporting
ReportOnEveryRetryAmount : retries X amounts, before
setting project to exception state, and set the counter back to 0
so you
will get reports every X integrations
Reporting means : running the publisher section, ever error will be
logged in the ccnet log no matter which
settings are choosen.
I know the email publisher must be updated also, so you have more control on
who gets the exception mails,
this is already on the todo list
http://jira.public.thoughtworks.org/browse/CCNET-1423
I think this covers all scenarios, does anybody see holes in this?
please comment, so we know what you want, before it is in production ;-)
with kind regards
Ruben Willems
---------- Forwarded message ----------
From: Ruben Willems <[email protected]>
Date: Thu, Mar 26, 2009 at 10:43 AM
Subject: Re: [ccnet-user] Re: sticky exception status
To: [email protected]
Hi all
just thought of something :
It seems that we are focussing mainly on interval triggers, (which is ok)
suppose there is a source control error with a schedule trigger (nightly
build),
what would the wanted behaviour be?
this has mainly to do how the retry will be implemented :
° wait 5 seconds and retry
or
° place project back in the Q and wait for next integration
with the interval trigger, both approaches are more or less the same
I prefer to place it back in the Q (basically it means ignore source control
exceptions until MaxSourceControlExceptions is reached)
this means that the waiting period is longer (trigger interval), 30 seconds
* 5 = 2.5 minutes that source control has to recuperate
if you have wait 5 seconds before retry, and it means source control has 5 *
5 = 25 seconds to recuperate
but with a schedule trigger, the next integration round may be : tomorrow
any thoughts on this?
or should we add MaxSourceControlExceptions on the trigger level ? (and when
not set, take it from the project level)
( just brainstorming here ... )
with kind regards
Ruben Willems
On Thu, Mar 26, 2009 at 8:54 AM, Ruben Willems <[email protected]>wrote:
> Hi
>
> That is even better, automatic recovery !
> +1 for me
>
> with kind regards
> Ruben Willems
>
> On Thu, Mar 26, 2009 at 8:43 AM, Per-Jonny Käck <[email protected]> wrote:
>
>> I also really like the idea of having 1 exception build.
>>
>> But with Craigs proposed solution I'm not sure I see the point of stopping
>> the project.
>> If the network/source control system comes up again it feels wasted that
>> the project has already been stopped.
>>
>> I think I would be happier with the following scenario.
>> - if get modifications fails, it only logs the error (similar to
>> pre-1.4.3).
>> - it counts the number of successive failures
>> - if the number of successive failures EQUALS the value in
>> maxSourceControlRetries then it will log an error, change the build status
>> to exception and CONTINUE RUNNING the project.
>> - if the above state occurs, it will also run the publishers to send out
>> any e-mails, notifications, etc.
>>
>> What happens when the project continue running after the failed build due
>> to maxSourceControlRetries?
>> - if get modifications fails, it continue forever to log the error
>> (similar to pre-1.4.3)
>> - it counts the number of successive failures up to Int32.MaxValue
>> - it should NOT reach maxSourceControlRetries again and not trigger any
>> new publishers.
>> - as soon as the get modifications succeeds and a "real build" is started
>> the exception count is reset again
>>
>> Comments?
>>
>> //P-J
>>
>> 2009/3/26 Ruben Willems <[email protected]>
>>
>> Hi Craig
>>>
>>> Sounds ok to me
>>>
>>> only a question :
>>> when is it retried ?
>>> immediately, or is the project placed back in the Q?
>>>
>>> if it is immediately, what would be the sleep/wait time before the retry
>>> occurs?
>>>
>>> I like the idea of having 1 exception build, even if the
>>> maxSourceControlRetry value is set to 5
>>> that is indeed a very great improvement over the current approach, where
>>> every attempt would run the publisher block.
>>>
>>>
>>> with kind regards
>>> Ruben Willems
>>>
>>>
>>>
>>> On Wed, Mar 25, 2009 at 6:59 PM, Craig Sutherland <
>>> [email protected]> wrote:
>>>
>>>>
>>>> Hello all,
>>>>
>>>> I've been looking into this issue and seeing how we can improve
>>>> things, since it obviously is a contentious issue.
>>>>
>>>> First, what we were trying to achieve: previously errors with source
>>>> control were ignored or they would just crash the server. Obviously
>>>> this state was not particularly good. The changes now include source
>>>> control operations errors as build exceptions. Unfortunately, this now
>>>> appears to have gone the other way and made CC.Net too sensitive to
>>>> issues with source control.
>>>>
>>>> So what I am planning on doing is adding some tolerance to the
>>>> process. There is already a project setting called
>>>> maxSourceControlRetries, which currently sets the allowed number of
>>>> source control failures before stopping the project.
>>>>
>>>> What I am going to do is modify the source control checking, so if get
>>>> modifications fails, it only logs the error (similar to pre-1.4.3).
>>>> However, it will count the number of failures - if the number of
>>>> failures reaches the value in maxSourceControlRetries (by default this
>>>> is five), then it will log an error, change the build status to
>>>> exception and stop the project. If this state occurs, it will also run
>>>> the publishers to send out any e-mails, notifications, etc.
>>>>
>>>> This way people now have some control over the sensitivity of source
>>>> control checks. If you want to completely ignore this feature, set
>>>> maxSourceControlRetries to some really high number (it's an Int32
>>>> though, so not too high), or if your source control/network/etc. is
>>>> really, really good and you don't except any problems, set it to a low
>>>> number (e.g. 0).
>>>>
>>>> What do people think about this approach?
>>>>
>>>>
>>>> Craig
>>>>
>>>> On Mar 26, 2:33 am, Jon W <[email protected]> wrote:
>>>> > On Tue, Mar 24, 2009 at 9:19 PM, si <[email protected]> wrote:
>>>> >
>>>> > >>> I now have 7 projects in "Exception" status (which are actually
>>>> > >>> running) with failed builds and none of them have actually failed,
>>>> > >>> they only failed because our server wasn't able to query svn
>>>> because
>>>> > >>> another build appears to have saturated available resources on the
>>>> > >>> server..
>>>> >
>>>> > >> ...and that's where our opinions differ: I see the situation as an
>>>> actual
>>>> > >> failure - if you aren't able to guarantee a successful build in a
>>>> timely
>>>> > >> fashion, it's a failure (along the lines of a typical "enterprise"
>>>> SLA)...
>>>> > >> just a question of filtering different failure notifications to
>>>> different
>>>> > >> people based on the cause. But then my opinion is just one of many,
>>>> and
>>>> > >> everyone is entitled to their own ;-)
>>>> >
>>>> > > Of course, and choice is good!
>>>> >
>>>> > > Anyway, I took Ruben's advice about setting up a single queue for
>>>> all
>>>> > > our projects, and that should resolve the problem.
>>>> >
>>>> > > I'm still not completely convinced that the underlying cause (svn
>>>> log
>>>> > > timeout) was due to the server or network, it may well be, but if
>>>> more
>>>> > > reports start coming in then there might be some other issue at
>>>> work.
>>>> >
>>>> > I agree with Si. I would like the svn timeouts to be logged just as
>>>> > they were pre-1.4.3, but I don't think the builds should enter the
>>>> > failed state due to a 'checking for modifications' action. An
>>>> > administrative notification can be sent, but that should only notify
>>>> > the administrator of the build system, and not the 100 developers who
>>>> > only need to focus on code modifications.
>>>> >
>>>> > Thanks,
>>>> > Jon
>>>>
>>>
>>>
>>
>
