On Wed, Jul 07, 2004 at 02:42:22PM -0400, Michael G Schwern wrote:
> On Fri, Jun 04, 2004 at 10:18:37PM -0400, Andrew Pimlott wrote:
> > I started using Test::Inline, and I have two related comments. (I hope
> > this is the right place to bring them up.)
> >
> > 1. I don't think that pod2test should do anything more than the minimum
> > to construct a valid test script. Anything else that is useful
> > should be part of the standard Test:: modules.
>
> pod2test is poorly architected but I don't see anything it does that
> I'd want in a module. What were you thinking of?
I was mostly thinking about the capturing of STDOUT and STDERR, but I'm
alsa suggesting it as a general principle. I guess the only other thing
is the automatic
use Test::More 'no_plan';
This is probably harmless, but on the other hand it's trivial to include
it yourself, so I don't see the point. Someday we will all be using
Test::Most instead. (waitaminnit... search.cpan.org... whew, not there
yet)
> > 2. I don't like it capturing STDOUT and STDERR when I don't ask for it.
> > I realize that well-behaved tests shouldn't write directly to
> > STDOUT, but since the point of tests is that code is not always
> > well-behaved, it's nice to get the clues that STDOUT and STDERR may
> > provide. There are of course better ways to capture this output
> > when that's desired.
>
> You did ask for it, you used Test::Inline. :)
No, you dunderhead, I used Test::Inline to include my tests in the same
file as my code, but separate enough that the code still works when the
tests don't compile, in a standard, well-supported way.
> Its rarely useful for a test to output to STDOUT. Can you think of a
> non-contrived example where it would be desirable? OTOH, example code
> often involves printing. So trapping STDOUT seems obvious to me.
Well, for one thing, printing directly to STDOUT is documented in Test
and Test::Tutorial. You may say, you're not supposed to be
sophisticated enough to use Test::Inline and simple enough to print the
results directly; but I don't see why the same test environment
shouldn't be supported everywhere. However, you're right that in
practice, I haven't missed seeing STDOUT.
I didn't think of your point about example code often printing, because
my test code is not (currently) documentation (and obviously I did not
read the Test::Inline documentation carefully). On the other hand,
there is generally no need to print in example code; assigning to a
variable will get the point across just as well.
> STDERR is another issue. Lots of interesting information might go out to
> STDERR during a failure that you'd like to see. But again, examples
> often call warn.
Missing STDERR was exceedingly annoying for me--at first because I
didn't understand why my test suddenly stopped with no warning, and
subsequently because I had to keep editing my test files in order to see
the missing diagnostics. I have since removed the capturing from my
copy of pod2test.
> Since trapping STDOUT/ERR is really only useful for example code, maybe
> T::I should stop doing it for test blocks.
This would remove my current pain, but ...
> That still leaves warnings in examples. One can write "is $__STDERR__, ''"
> but it would happen at every example block and that gets tedious. Perhaps
> T::I can get smart. If $__STDERR__ is not looked at in the example_testing
> block an implied "is $__STDERR__, ''" is run. Simple enough since
> $__STDERR__ is a tied variable.
This doesn't handle the most important case: What if the example code
died? You could eval the example code; however every complication you
add risks adding confusion and interfering with the tests themselves.
Another case: there is a warning in the example, and a test failure
in the example_testing, where the warning contained clues about what led
to the failure. In your scheme, the warning won't be seen.
I think it would be best to make people who want STDOUT and/or STDERR
captured ask for it, on a per-example basis, in the POD. I don't know
how you'd fit this into POD syntax, however.
Here is another approach: Import a special "print" and "warn" into main
when running examples, which capture their arguments and don't produce
any output. This wouldn't handle your getprint example, but it's
probably a 95% solution.
> Also, --no-trap-stdout and --no-trap-stderr options might be in order.
I would much rather encode these directives in the POD than in the
command line.
Andrew
