On 11/13/06, Joachim Draeger <[EMAIL PROTECTED]> wrote:
Am Montag, den 13.11.2006, 12:00 +0100 schrieb Bernd Fondermann:
> > IMO a failing test is as valuable as a passing one. Maybe even more
> > because it reminds us to do something.
>
> > I don't think that it is an indicator of quality to have always 100% of
> > tests passing.
>
> My unit test 1x1 says: test runs have a binary result. It is either
> green or red. If red, you have to fix the test. If you do nothing, the
> test will soon be accompanied by a second failing test. Nobody checks
> for failing tests anymore.
I do not propose to do nothing when a test changes from green to red. I
propose committing tests that fail because you don't have a solution so
far.
I can't see the advantage in accepting only passing tests.
Because thats the fundamental unit test paradigm. The whole red/green
thing is build on this.
If you know something fails, you write a test testing exactly this and
document it in code + in JIRA.
> That does not necessarily mean every failing tests is a subject to
> immediate fix. For example, this is not possible in
> test-driven-development which is based on failing tests.
Would you accept failing tests used for TDD in James?
As long as you are developing locally, yes.
Before the commit, all tests should pass. If work is resumed next day
and the feature gets implemented, the test magically gets to your
attention because it is now failing and gets inverted. No searching in
large sections of commented or uncommitted code. It is very
straightforward.
(Side note: Some unit test experts even say that new tests should be
written to fail once and changed to succeed afterwards. Only this way
it is proven that the test is able to detect a breakage of the thing
it is testing.)
> But intended-to-fail failing tests obscures all
> oops-something-bad-happened failing tests.
I think that is only a management/administrative issue. If there are a
lot of tests we need a workflow not to oversee any changes in the
results.
And the only way IMHO to achieve this is to be green all (read: most)
of the time.
Do you refer to psychological aspects? I consider the message "don't
start working on a test, it could fail" more bad than "there is one
failing test, nobody would mind a second...".
If somebody starts working on tests, they _will_ fail at the
beginning. But go away with a failing test and do no clean ups is not
polite.
Just like checking in non-compiling code. You could say, well it is
not compiling, never mind. I get back to that sometime.
> > There often are known bugs for a long time, some of them not even
> > documented anywhere.
>
> Agreed. It is a good thing to have tests that document
> bugs/errors/missing features/evolving code.
> But these test should not be failing, be commented out or be contained
> in the a separate "failing-tests-suite".
> They should _succeed_, with a comment pointing to the JIRA they
> reproduce and document.
> When such an issue gets fixed, the test fails, will be detected and
> the assertion in question can be inverted. Voila.
And when you have an Exception you add a try/catch? Who guarantees that
nobody forgets what is right?
By adding a
fail("this fails as soon as JIRA-9999 is fixed");
following the statement raising the exception in the try block.
As soon as the fix is implemented this "fail" is detected and removed
and the catch block is changed to contain
fail("no longer raising exception since JIRA-9999 is fixed);
I never heard of this as a good practice for unit tests. IMO *this*
would obscure the tests.
The assertion logic could be more complicated than a single boolean
assert. (which could be an anti-pattern of course)
But no so complicated to justify polluting the whole test suite with failures.
The developer working on the fix runs the tests again and again, maybe
using a debugger. I think this should be done on the original, not on a
inverted one.
Agreed. But you are talking about work in progress on a working copy.
Unit tests can be part of the definition. They should not be changed in
any way as long they are valid.
Not agreed. This is a too dogmatic point of view.
>From the junit FAQ (okay some kind of TDD related): "When all the tests
pass, you know you're done!"
+1
Before changing/inverting anything I would comment out them completely.
Many people do, I think it has downsides.
> > I propose to accept failing tests in the codebase. Nightly build should
> > not fail just because tests are failing. Instead we could provide a
> > testreport in the same folder.
> > Of course tests ran from ant should not be interrupted by the first
> > failure. (haltonfailure="no")
> +1
Maybe we could start a vote on that after discussing.
If there is nobody objecting it would be sufficient to go without a formal vote.
> > Well, of course this brings some administrative overhead. Maybe we need
> > a solution to sort our tests using TestSuites.
> > We could separate the tests known to fail to be warned at once when a
> > change brings additional tests to fail.
> -1
Why? Doesn't this avoid the obscuring issue? The regular tests pass and
you have a binary result. Wouldn't it be nice to have all known-bugs in
one TestSuite?
From my experience, TestSuites get outdated very fast because
maintaining them is administrative overhead. I do not use them
anymore.
> > I don't consider slow tests as bad tests. They are just inappropriate
> > for a ad hoc results. Even if they took an hour it's okay, but we need
> > to find a way not to force people to run them all the time.
>
> Not agreed. According to my experience unit tests should be fast and
> easy. Otherwise your unit test setup is flawed or could be simplified.
I agree. That's why I tried to avoid the term "unit test". :-)
> Other tests, of course, for example integration or compliance test
> could take much more time.
Right. As I started to test which IMAP commands work in the current
implementation I wrote integration tests. These have proven to be very
useful.
If the problem arises, that they are to slow for "every-day-use" we
should separate, but not change them.
integration tests don't belong into a unit test suite.
per definition, in java, unit tests test (more or less) a single class
("the unit").
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
