If you reply to this, please make sure you reply to the 2 cases
involving the dog, this is my main objection to using TODO tests in
this manner.
On 02/12/2007, Michael G Schwern <[EMAIL PROTECTED]> wrote:
> Fergal Daly wrote:
> > One of the supposed benefits of using TODO is that you will notice
> > when the external module has been fixed. That's reasonable but I don't
> > see a need to inflict the confusion of unexpectedly passing tests on
> > all your users to achieve this.
>
> Maybe we should just change the wording and presentation so we're not
> inflicting so much.
>
> Part of the problem is it screams "OMG! UNEXPECTEDLY SUCCEEDED!" and the user
> goes "whoa, all caps" and doesn't know what to do. It's the most screamingest
> part of Test::Harness 2.
>
> Fortunately, Test::Harness 3 toned it down and made it easier to identify
> them.
>
> Test Summary Report
> -------------------
> /Users/schwern/tmp/todo.t (Wstat: 0 Tests: 2 Failed: 0)
> TODO passed: 1-2
>
> TAP::Parser also has a "todo_passed" test summary method so you can
> potentially customize behavior of passing todo tests at your end.
>
>
> I agree with Eric, these tests are extra credit. "Unexpectedly working
> better" != "failure" except in the most convoluted situations. Their
> intention is to act as an alternative to commenting out a test which you can't
> fix right now. An executable TODO list that tells you when you're done, so
> you don't forget.
>
> It should not halt installation, nothing's wrong as far as the user's
> concerned. However, it does mean "investigate" and it would be nice if this
> information got back to the author. It would be nice if CPAN::Reporter
> reported passing TODO tests... somehow.
>
>
> > Another downside of using TODO like this is that when the external
> > module is fixed, you have to release a new version of your module with
> > the TODOs removed. These tests will start failing for anyone who
> > upgrades your module but not broken one but in reality nothing has
> > changed for that user,
>
> As long as you're releasing a new version, why would you not upgrade your
> module's dependency to use the version that works?
Your module either is or isn't usable with version X of Foo.
If it is usable then you would not change your dependency before or
after the bug in version X is fixed (maybe I have a good reason not to
upgrade Foo and you wouldn't want your module to refuse to install if
it is actually usable).
If it isn't usable then marking your tests as TODO was the wrong thing
to do in the first place, you should have bailed out due to
incompatibility with version X and not bothered to run any tests at
all. I think Extutils::MM does not have any way to specify complex
version dependencies but with Module::Build you could say
requires => {
'Foo' => ' < X '
}
and you leave your tests untouched. When Foo X+1 comes out, you just
update your requires to say Foo => '!X' and everybody's happy.
I am basically against the practice of using TODO to cope with
external breakage. Not taking unexpected passes seriously encourages
this practice. Apart from there being other ways to handle external
breakage that seem easier, using TODO is actually dangerous as it can
cause false passes in 2 ways. Says version X of Foo has a non-serious
bug so you release version Y of Bar with some tests marked TODO. The
we risk
1 Version X+1 of Foo is even worse and will cause Bar to eat your dog.
Sadly for your dog, the test that might have warned him has been
marked TODO.
2 You're using version X-1 of Foo, everything is sweet, your dog can
relax. You upgrade to version Y+1 of Bar which has a newly introduced
dog-eating bug. This bug goes undetected because the tests are marked
TODO. So long Fido.
I still have not seen an example of using TODO in this manner that
isn't better handled in a different way.
As before, I am not advocating changing the current Test::* behaviour
to fail on unexpected passes as that would just be a mess. It's just
that whenever this is discussed it ends up with people advocating what
I consider wrong and dangerous uses of TODO and so I am pointing this
out again,
F
