Test::Buildder v. Test2 subtests

[Ricardo Signes]

I've hit a nasty (to me) difference between Test2::API::run_subtest and
Test::Builder::subtest.  Shout out to Matthew Horsfall for helping localize the
problem.  Here's my trivial reproducer:

  #!perl
  use strict;
  use warnings;
  use Test::More;
  use Test2::API;

  my $code = sub {
    if (fork) {
      wait;
      cmp_ok($?, '!=', 0, "subprocess died");
    } else {
      die "123";
    };
  };

  my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
  $call->(test => $code);
  done_testing;

When running with Test::More...

  ~$ perl foo 1
  # Subtest: test
  123 at foo line 12.
      ok 1 - subprocess died
      1..1
  ok 1 - test
  1..1

The die in the subprocess causes the child process to exit nonzero and the
tests run normally.  Great!  This is how my tests work.  When I switch them
to Test2::API::run_subtest, though, I get this:

  ~$ perl foo 0
  # test
  not ok 1 - test
  # Failed test 'test'
  # at foo line 17.
  # Caught exception in subtest: 123 at foo line 12.
  1..1
      not ok 1 - subprocess died
      #   Failed test 'subprocess died'
      #   at foo line 10.
      #          got: 0
      #     expected: anything else
      1..1
  not ok 1 - test
  # Failed test 'test'
  # at foo line 17.
  1..1
  # Looks like you failed 1 test of 1.

In the forked code, the die causes the subtest to end in failure *in the fork*
meaning we get a repeat result for test 1: one from the child, one from the
parent.  Also, the child is now exiting zero instead of nonzero, despite the
fact that it threw an exception.

It gets worse [for me]!  Above, I had a cmp_ok on $? that failed, so I had
duplicate *not ok* results for test 1.  If I replace that cmp_ok with a pass, I
get duplicate *conflicting* results for test 1... and then the main process
terminates zero.  This means that the test seems to pass if you look at its $?,
but the harness sees it as a failure because its total output does include
failures (and non-increasing test numbers).

Adding Test2::IPC does not help.  The issue is not synchronising test counters,
or the like.  It's that I don't want the subprocess's exit to cause a subtest
exit in the parent process.  I think that's right: the only process that should
turn exceptions into subtest fails is the process that created the subtest.

a) am I wrong?
b) advice on how to procede?

-- 
rjbs

signature.asc
Description: Digital signature