Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Michael G Schwern
Chris Dolan wrote: > On Dec 2, 2007, at 1:34 PM, nadim khemir wrote: >> Because a TODO means that it is not done not: it might happend to be >> done but >> I'm not really sure, maybe I get lucky. >> >> Either one removes the TODO and all is fine. Or it might just be a >> side effect >> that you ha

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Chris Dolan
On Dec 2, 2007, at 4:11 PM, Michael G Schwern wrote: Fergal Daly wrote: Another downside of using TODO like this is that when the external module is fixed, you have to release a new version of your module with the TODOs removed. These tests will start failing for anyone who upgrades your mod

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Chris Dolan
On Dec 2, 2007, at 1:34 PM, nadim khemir wrote: Because a TODO means that it is not done not: it might happend to be done but I'm not really sure, maybe I get lucky. Either one removes the TODO and all is fine. Or it might just be a side effect that you haven't planned that makes the test

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Michael G Schwern
Fergal Daly wrote: >> As long as you're releasing a new version, why would you not upgrade your >> module's dependency to use the version that works? > > Your module either is or isn't usable with version X of Foo. > > If it is usable then you would not change your dependency before or > after the

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Fergal Daly
If you reply to this, please make sure you reply to the 2 cases involving the dog, this is my main objection to using TODO tests in this manner. On 02/12/2007, Michael G Schwern <[EMAIL PROTECTED]> wrote: > Fergal Daly wrote: > > One of the supposed benefits of using TODO is that you will notice >

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Michael G Schwern
Fergal Daly wrote: > One of the supposed benefits of using TODO is that you will notice > when the external module has been fixed. That's reasonable but I don't > see a need to inflict the confusion of unexpectedly passing tests on > all your users to achieve this. Maybe we should just change the

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Fergal Daly
http://www.mail-archive.com/perl-qa@perl.org/msg06865.html has the previous round on this topic. My memory is hazy but my view was that people are using TODO in strange ways and making this a failure would break that. The strange way I remember (and has been brought up again by Chris Dolan) is rel

Re: shouldn't "SKIPPED" mean failure?

2007-12-02 Thread Eric Wilhelm
# from Paul Johnson >How is "didn't do what I expected" *ever* any sort of success? It *did* do what the *test* expected ("ok(1)"). Yes, the TODO expected it to fail, but if it is passing, you have "more than success". It might need attention, but "failure" is a subset of "needs attention" or

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Eric Wilhelm
# from nadim khemir # on Sunday 02 December 2007 11:34: >> How is "extra credit" *ever* any sort of failure? > >Because a TODO means that it is not done not:  it might happend to be > done but I'm not really sure, maybe I get lucky. No, the latter is almost exactly what "todo" means. More precis

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread nadim khemir
On Sunday 02 December 2007 18:51, Eric Wilhelm wrote: > # from Ovid > > # on Sunday 02 December 2007 07:47: > >--- nadim khemir <[EMAIL PROTECTED]> wrote: > >> The subject says it all. IE: > > ... > > >> t/20_policies.t 152 578 583 > > > >It just means "you need to investigate th

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Paul Johnson
On Sun, Dec 02, 2007 at 09:51:49AM -0800, Eric Wilhelm wrote: > How is "extra credit" *ever* any sort of failure? How is "didn't do what I expected" *ever* any sort of success? Just playing devil's advocate here really, but experience has taught me to be rather conservative when it comes to test

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Chris Dolan
On Dec 2, 2007, at 9:37 AM, nadim khemir wrote: The subject says it all. IE: All tests successful (2 subtests UNEXPECTEDLY SUCCEEDED), 7 tests skipped. Passed TODO Stat Wstat TODOs Pass List of Passed -- - t/2

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Eric Wilhelm
# from Ovid # on Sunday 02 December 2007 07:47: >--- nadim khemir <[EMAIL PROTECTED]> wrote: >> The subject says it all. IE: ... >> t/20_policies.t 152 578 583 > >It just means "you need to investigate this further". Personally, I >would like to see it optionally mean failure,.

Re: TAP::Builder

2007-12-02 Thread Ovid
--- Ovid <[EMAIL PROTECTED]> wrote: > Side note: those features I really want control over in > Test::Harness > are the plan() and ok() methods. There's no clean way for me to do > that. Just look at the constructor: > > my $Test = Test::Builder->new; > sub new { > my($class) = shift

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread nadim khemir
On Sunday 02 December 2007 16:47, Ovid wrote: > --- nadim khemir <[EMAIL PROTECTED]> wrote: > > The subject says it all. IE: > > > > All tests successful (2 subtests UNEXPECTEDLY SUCCEEDED), 7 tests > > skipped. > > Passed TODO Stat Wstat TODOs Pass List of Passed > > -

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread Ovid
--- nadim khemir <[EMAIL PROTECTED]> wrote: > The subject says it all. IE: > > All tests successful (2 subtests UNEXPECTEDLY SUCCEEDED), 7 tests > skipped. > Passed TODO Stat Wstat TODOs Pass List of Passed > --- > t/20_poli

Re: TAP::Builder

2007-12-02 Thread Ovid
--- Michael G Schwern <[EMAIL PROTECTED]> wrote: > This is easier to do with Test::Builder because custom features meant > custom > functions. Custom functions avoid functionality overlap and users > write code > specifically for them. Test::Harness and prove have no such luxury. Right, let's t

shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

2007-12-02 Thread nadim khemir
The subject says it all. IE: All tests successful (2 subtests UNEXPECTEDLY SUCCEEDED), 7 tests skipped. Passed TODO Stat Wstat TODOs Pass List of Passed --- t/20_policies.t 152 578 583 (nice report