Actually at first glance a possible problem is this: You are using 2
functions that are not intended to be interchangeable.
Test::More::subtest() is a tool intended to run a subtest.
Test2::API::run_subtest() is a helper intended to write tools like
'subtest()'.
I would like to see the results when you use
Test2::Tools::Subtest::subtest_streamed() which is the actual equivalent to
Test::More::subtest().
-Chad
On Tue, Dec 13, 2016 at 8:09 AM, Chad Granum <exodi...@gmail.com> wrote:
> I will have to look into this when I have more time.
>
> On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <perl...@rjbs.manxome.org>
> wrote:
>
>> I've hit a nasty (to me) difference between Test2::API::run_subtest and
>> Test::Builder::subtest. Shout out to Matthew Horsfall for helping
>> localize the
>> problem. Here's my trivial reproducer:
>>
>> #!perl
>> use strict;
>> use warnings;
>> use Test::More;
>> use Test2::API;
>>
>> my $code = sub {
>> if (fork) {
>> wait;
>> cmp_ok($?, '!=', 0, "subprocess died");
>> } else {
>> die "123";
>> };
>> };
>>
>> my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
>> $call->(test => $code);
>> done_testing;
>>
>> When running with Test::More...
>>
>> ~$ perl foo 1
>> # Subtest: test
>> 123 at foo line 12.
>> ok 1 - subprocess died
>> 1..1
>> ok 1 - test
>> 1..1
>>
>> The die in the subprocess causes the child process to exit nonzero and the
>> tests run normally. Great! This is how my tests work. When I switch
>> them
>> to Test2::API::run_subtest, though, I get this:
>>
>> ~$ perl foo 0
>> # test
>> not ok 1 - test
>> # Failed test 'test'
>> # at foo line 17.
>> # Caught exception in subtest: 123 at foo line 12.
>> 1..1
>> not ok 1 - subprocess died
>> # Failed test 'subprocess died'
>> # at foo line 10.
>> # got: 0
>> # expected: anything else
>> 1..1
>> not ok 1 - test
>> # Failed test 'test'
>> # at foo line 17.
>> 1..1
>> # Looks like you failed 1 test of 1.
>>
>> In the forked code, the die causes the subtest to end in failure *in the
>> fork*
>> meaning we get a repeat result for test 1: one from the child, one from
>> the
>> parent. Also, the child is now exiting zero instead of nonzero, despite
>> the
>> fact that it threw an exception.
>>
>> It gets worse [for me]! Above, I had a cmp_ok on $? that failed, so I had
>> duplicate *not ok* results for test 1. If I replace that cmp_ok with a
>> pass, I
>> get duplicate *conflicting* results for test 1... and then the main
>> process
>> terminates zero. This means that the test seems to pass if you look at
>> its $?,
>> but the harness sees it as a failure because its total output does include
>> failures (and non-increasing test numbers).
>>
>> Adding Test2::IPC does not help. The issue is not synchronising test
>> counters,
>> or the like. It's that I don't want the subprocess's exit to cause a
>> subtest
>> exit in the parent process. I think that's right: the only process that
>> should
>> turn exceptions into subtest fails is the process that created the
>> subtest.
>>
>> a) am I wrong?
>> b) advice on how to procede?
>>
>> --
>> rjbs
>>
>
>
