I will have to look into this when I have more time.
On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <perl...@rjbs.manxome.org>
wrote:
> I've hit a nasty (to me) difference between Test2::API::run_subtest and
> Test::Builder::subtest. Shout out to Matthew Horsfall for helping
> localize the
> problem. Here's my trivial reproducer:
>
> #!perl
> use strict;
> use warnings;
> use Test::More;
> use Test2::API;
>
> my $code = sub {
> if (fork) {
> wait;
> cmp_ok($?, '!=', 0, "subprocess died");
> } else {
> die "123";
> };
> };
>
> my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
> $call->(test => $code);
> done_testing;
>
> When running with Test::More...
>
> ~$ perl foo 1
> # Subtest: test
> 123 at foo line 12.
> ok 1 - subprocess died
> 1..1
> ok 1 - test
> 1..1
>
> The die in the subprocess causes the child process to exit nonzero and the
> tests run normally. Great! This is how my tests work. When I switch them
> to Test2::API::run_subtest, though, I get this:
>
> ~$ perl foo 0
> # test
> not ok 1 - test
> # Failed test 'test'
> # at foo line 17.
> # Caught exception in subtest: 123 at foo line 12.
> 1..1
> not ok 1 - subprocess died
> # Failed test 'subprocess died'
> # at foo line 10.
> # got: 0
> # expected: anything else
> 1..1
> not ok 1 - test
> # Failed test 'test'
> # at foo line 17.
> 1..1
> # Looks like you failed 1 test of 1.
>
> In the forked code, the die causes the subtest to end in failure *in the
> fork*
> meaning we get a repeat result for test 1: one from the child, one from the
> parent. Also, the child is now exiting zero instead of nonzero, despite
> the
> fact that it threw an exception.
>
> It gets worse [for me]! Above, I had a cmp_ok on $? that failed, so I had
> duplicate *not ok* results for test 1. If I replace that cmp_ok with a
> pass, I
> get duplicate *conflicting* results for test 1... and then the main process
> terminates zero. This means that the test seems to pass if you look at
> its $?,
> but the harness sees it as a failure because its total output does include
> failures (and non-increasing test numbers).
>
> Adding Test2::IPC does not help. The issue is not synchronising test
> counters,
> or the like. It's that I don't want the subprocess's exit to cause a
> subtest
> exit in the parent process. I think that's right: the only process that
> should
> turn exceptions into subtest fails is the process that created the subtest.
>
> a) am I wrong?
> b) advice on how to procede?
>
> --
> rjbs
>
