From: Michael G Schwern <[EMAIL PROTECTED]>
> print "1..1\n";
> print "not ok 1\n";
> print STDERR "# Your shit is broke.\n";
>
> How will TAPx::Parser deal with that?
You know, you could skip this entire response and read my last couple of
paragraphs. That will sum it up.
Hmm, let's see:
TAPx-Parser $ tprove test.pl
# Your shit is broke.
test....not ok.
Failed: 1
Tests run: 1
Passed: 0
Failed: 1
Errors: 0
1 wallclock secs ( 0.01 usr 0.00 sys + 0.03 cusr 0.01 csys = 0.05 CPU)
And let's see what prove does:
TAPx-Parser $ prove test.pl
test....# Your shit is broke.
test....FAILED test 1
Failed 1/1 tests, 0.00% okay
Failed Test Stat Wstat Total Fail List of Failed
-------------------------------------------------------------------------------
test.pl 1 1 1
Failed 1/1 test scripts. 1/1 subtests failed.
Files=1, Tests=1, 0 wallclock secs ( 0.01 cusr + 0.01 csys = 0.02 CPU)
Failed 1/1 test programs. 1/1 subtests failed.
Well, that leaks through with both tprove and prove. So I've apparently not
*lost* any behavior.
> Test::Builder is not the only game in town. You
> can't patch the world's tests for your convenience.
I have no intention to. I'm optimizing for the common case and if people use
that case (Test::Builder based tools), they know that, for the time being, they
get behavior more in line with what some folks want.
> Here's another thing which will break:
>
> use Test::More tests => 2;
>
> is( "foo", "bar" );
> TODO: {
> local $TODO = "my shit is broke";
> is(23, 42);
> }
>
> The user should not see the diagnostics from that TODO failure.
Agreed. That's one weak point. One could possibly apply the heuristic of
"swallow comments after a failed TODO" or one could say 'hey, the tests
passed.' This is one difference for which there is no easy solution with the
current state of TAP. Fortunately, I report the test results correctly.
If one chose to swallow comments after TODO failures, one might, on non-verbose
mode, lose a user-supplied diag comment if it came immediately after the TODO
statement. No other failure comments would be lost since those would come
after a test. But if I was just to spit out the TODO diagnostics, how would
this break things? I'd imagine that any tool that chooses to enjoy the
benefits of TAPx::Parser would know that it's using TAPx::Parser.
> Another really bad bad no no no awful thing about
> this scheme is the parser is changing the behavior of the tests.
> I run it in the parser and I get one behavior. I run it outside the
> parser and I get another behavior. This creates heisenbugs.
> This is a Very Bad Thing.
NO IT DOES NOT!!!!! That's FUD, plain and simple. This scheme in no way,
shape, or form changes the behavior of tests. It correctly reports how many
tests passed. It correctly reports how many tests failed. It correctly
reports how many tests were skipped. It correctly reports how many tests were
TODO. It correctly reports how many tests unexpectedly succeeded. It also
does this far more flexibly than Test::Harness does.
However, because of a design limitation in TAP and how Test::Simple was
implemented, we get one tiny *VISUAL* change. We don't get different behavior
in tests. We get one difference in behavior in their presentation. We see if
they succeed or fail.
Oh, wait! We get more than one difference in behavior. Because STDERR and
STDOUT get synched, this never happens again:
ok 15 - ... and splitting the sql should succeed
# @records = (
ok 16 - ... but the original SQL should remain unchanged
# 'alter table test_colors add column foo varchar(200)'
# );
not ok 17 - We should be able to "split" a single statement
The TODO bit doesn't happen to me too much. The above happens to me every day
(I'm not making this up) and it would be nice if it never happened again.
> These are all Very Bad Things to be putting in so early in the
> design. They're not solutions, they're hacks. The parser should
> not change the behavior of the tests.
It doesn't change the behavior of tests. It changes the *output* of tests.
Please stop whipping out the bogeyman to scare the children.
> > 2. So many people have contacted me to let me know they want to use
> > TAPx::Parser.
> #2... How does not capturing STDERR stop people
> from using TAPx::Parser? If it did we couldn't use
> Test::Harness. This is what I don't understand. Where's
> the urgency here? I mean, its a nice feature but how
> is it a show stopper?
This is a feature people have asked for. They want it. They'd like to click on
failed test in a GUI and have all of the comments before and after that tests
spit out. It's not perfect, it still requires human interpretation, but so
does our current system.
Plus, to be really blunt, if I wait for everyone else's schedules, I wouldn't
expect a TAP parser for a couple of years, if ever. That's not meant to be an
insult, either. You folks have lives. You have other projects you're working
on. I simply happen to have spare time and I'm going to use it. Later, I
might not have that spare time (how many times have we lost an open-source
developer to Everquest or a new girlfiend, for example?)
> Finally, I'm really not convinced the sync'd two
> stream issue can't be solved or that it hasn't been
> solved. Give it a little time. And again, has
> anyone talked to the CPANPLUS folks? I remember
> Jos going through just this sort of thing.
I'll shoot him an email.
As a final comment: I remember one of Adam Kenned's first posts about PPI (I
can't find it right now, though). I don't think he called it PPI back then,
but Randal called him on the carpet: "OMG, YOU CAN'T PARSE PERL!"
You know what? Randal was right. He's still right. But Adam ignored that and
went ahead and did it anyway.
Adam's solution is not perfect, but it's pretty damned good and a lot of folks
are building tools around it. They're building USEFUL tools around Adam's
imperfect, broken, impossible solution.
Yes, what I'm trying to do is hard, but if I wait until TAP 2.0, Perl 6 will be
out of beta by then. So I'm going to try and make it happen. You can tell me
that it's too hard or that I can't solve the TODO problem and I'll say "yup,
but I've got a 95% solution and I can live with that." People are asking for
this (yes, including the STDERR problem), so I'm going to deliver.
Cheers,
Ovid
