Actually at first glance a possible problem is this: You are using 2 functions that are not intended to be interchangeable.
Test::More::subtest() is a tool intended to run a subtest. Test2::API::run_subtest() is a helper intended to write tools like 'subtest()'. I would like to see the results when you use Test2::Tools::Subtest::subtest_streamed() which is the actual equivalent to Test::More::subtest(). -Chad On Tue, Dec 13, 2016 at 8:09 AM, Chad Granum <exodi...@gmail.com> wrote: > I will have to look into this when I have more time. > > On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <perl...@rjbs.manxome.org> > wrote: > >> I've hit a nasty (to me) difference between Test2::API::run_subtest and >> Test::Builder::subtest. Shout out to Matthew Horsfall for helping >> localize the >> problem. Here's my trivial reproducer: >> >> #!perl >> use strict; >> use warnings; >> use Test::More; >> use Test2::API; >> >> my $code = sub { >> if (fork) { >> wait; >> cmp_ok($?, '!=', 0, "subprocess died"); >> } else { >> die "123"; >> }; >> }; >> >> my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest; >> $call->(test => $code); >> done_testing; >> >> When running with Test::More... >> >> ~$ perl foo 1 >> # Subtest: test >> 123 at foo line 12. >> ok 1 - subprocess died >> 1..1 >> ok 1 - test >> 1..1 >> >> The die in the subprocess causes the child process to exit nonzero and the >> tests run normally. Great! This is how my tests work. When I switch >> them >> to Test2::API::run_subtest, though, I get this: >> >> ~$ perl foo 0 >> # test >> not ok 1 - test >> # Failed test 'test' >> # at foo line 17. >> # Caught exception in subtest: 123 at foo line 12. >> 1..1 >> not ok 1 - subprocess died >> # Failed test 'subprocess died' >> # at foo line 10. >> # got: 0 >> # expected: anything else >> 1..1 >> not ok 1 - test >> # Failed test 'test' >> # at foo line 17. >> 1..1 >> # Looks like you failed 1 test of 1. >> >> In the forked code, the die causes the subtest to end in failure *in the >> fork* >> meaning we get a repeat result for test 1: one from the child, one from >> the >> parent. Also, the child is now exiting zero instead of nonzero, despite >> the >> fact that it threw an exception. >> >> It gets worse [for me]! Above, I had a cmp_ok on $? that failed, so I had >> duplicate *not ok* results for test 1. If I replace that cmp_ok with a >> pass, I >> get duplicate *conflicting* results for test 1... and then the main >> process >> terminates zero. This means that the test seems to pass if you look at >> its $?, >> but the harness sees it as a failure because its total output does include >> failures (and non-increasing test numbers). >> >> Adding Test2::IPC does not help. The issue is not synchronising test >> counters, >> or the like. It's that I don't want the subprocess's exit to cause a >> subtest >> exit in the parent process. I think that's right: the only process that >> should >> turn exceptions into subtest fails is the process that created the >> subtest. >> >> a) am I wrong? >> b) advice on how to procede? >> >> -- >> rjbs >> > >