Hi Heiner,
Jens-Heiner Rechtien schrieb:
Jörg Jahnke wrote:
Ause just informed me about another solution that might remove the need
to have the test run on every CWS i.e. we wouldn't need to have the
tests mandatory. His idea is to run the tests on the Master Workspace
prior to announcing the CWS as "ready for CWS use". If a test fails then
this would result in a P1 issue that has to be fixed before the MWS can
be used by everyone. Very similar to how we handle it for the Smoketest
on the MWS nowadays.
...
Does that make sense?
Well, this moves the burden of hunting down which of the 40 or so
simultaneously integrated CWSs is responsible for the regression down on
RE, usually even with a considerable time pressure (you don't want to
know how many times Kurt and I have been asked when m212 will be ready).
That's absolutely not the idea behind all this.
That's why I mentioned a shortcut that allows for simply deactivating a
failed test on the master to get the necessary time to fix.
We know that the cost of a bug is smallest if found as early as
possible. The best thing is if the responsible developer finds it. The
developer usually has an immediate idea where to look if something
happens and there is no need for having the considerable overhead of
filing and tracking a specific bug.
Agreed - and just the reason why my initial proposal in the Wiki page
was the approach to do the testing on every CWS. But now many people
have uttered concerns about new mandatory tests being an unnecessary
hurdle for new code contributions. I just meant to bring in a new idea.
One of the better way of ensuring that as many bugs can be found as
early as possible are regression test. For this to work well four things
need to be ensured:
0) The regression test must be reasonable. This means it must be easy to
start and must finish in an acceptable time. After it's finished it must
be immediately clear if it was successful or if it failed.
The question is whether we will be able to offer tests that finish in an
"acceptable" time - whatever acceptable means. Currently it looks like
we can offer a set of tests that offer a reasonable code-coverage and
which take a few hours to run. To many here this does not seem
acceptable, for others even a larger amount of time seems to be.
What if the common agreement should say that the hours the tests take
are not an acceptable time. Would then my above proposal be better than
having no more regression tests at all? Of course for the quality of the
code it is best if we would do the additional regression tests the way
you outlined it and as it is mentioned in the Wiki page. But what if
that model is rejected as the time for the tests seems unacceptable?
Would we rather leave things as they are now or should we rather fall
back on the proposal I made above?
Regards,
Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
