Joachim Draeger wrote:
Am Montag, den 13.11.2006, 12:58 +0100 schrieb Stefano Bagnara:
But intended-to-fail failing tests obscures all
oops-something-bad-happened failing tests.
Right. This is the BIG problem of failing tests.
Okay, but I don't think that not writing, not executing or inverting
them is better.
I think tests should be written and executed. But if someone is
committing code that would break the tests he should simply wait to
commit. As soon as also the tests will work he will commit.
Simple and easy. We are not using test driven development, so I think
this is the simple way to go.
I have a lot of local branches of v2.3 and trunk, with various work in
progress. I synchronize them often with the online repository. If I run
tests and they fails I start looking at my code to understand what I
broke and how: if other people commit code that break the tests I will
loose much more time verifying where is the problem.
I can't have an in-memory list of currenlty non passing tests and the
cause. We have thousands of tests it would be a pain in the ass.
So as long as we are many to work on the same project we should take
care to have passing tests. Tests that fails means that something urgent
has to be done soon and it will probably stop others from working.
Imho it is the same of committing code that do not compile: if anyone
commit code that do not compile anyone having updated the localcopy will
have to take care of reverting the update in order to be able to keep
working.
It seems that everyone has its own argument: maybe we should simply
start a vote to decide how to keep working.
I propose to accept failing tests in the codebase. Nightly build should
not fail just because tests are failing. Instead we could provide a
testreport in the same folder.
Of course tests ran from ant should not be interrupted by the first
failure. (haltonfailure="no")
+1
Imho the nighly build should not be published if it does not pass tests.
Non passing tests should be only there by mistake, so the mistake could
have compromised the quality of the code unintentionally.
Everyone knows that nightly build is potentially unstable. I don't think
that the bug added by mistake, that brings a test to fail, is more
dangerous than the bug added by mistake that noone notices.
I find it more useful to encourage people submitting bug-exposing
failing tests and make everyone aware of these known bugs.
It's a matter of transparency.
Imho having failing tests creates more problems than the ones that solves.
Well, of course this brings some administrative overhead. Maybe we need
a solution to sort our tests using TestSuites.
We could separate the tests known to fail to be warned at once when a
change brings additional tests to fail.
-1
I don't understand why we should complicate things so much.
If after a change an assertTrue fail and is known to fail because the
test is wrong then it should be changed to assertFalse and a JIRA issue
added if the test need more attention or have to be completed.
Of course a test should be fixed if it is wrong. I'm not talking about
executing invalid tests. That's another story.
I mean tests that expose known bugs that won't be fixed in a short time
frame.
Joachim
If we want to start using failing test for a specific goal then let's
vote it and everyone will be aware of this. Otherwise tests must pass
and if they fail everyone will start loosing time to understand what he
did wrong on the code.
My personal preference is to avoid as hell committing code that will
make tests to fail. As like as committing code that does not compile or
run: of course the last one is the most difficult to be determined but
if it happens (and it happened to me many times) it should be handled as
high priority. We can make (by discussing) exception to this "rule" on
special case where the fix is more urgent than having passing tests, but
in trunk I give much more importance to avoid to loose committers time
than to provide fast fixes.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
