Created https://github.com/Test-More/test-more/issues/616 to track this where more people can see it.
On Mon, Dec 21, 2015 at 8:58 PM, Chad Granum <exodi...@gmail.com> wrote: > > > On Mon, Dec 21, 2015 at 5:10 PM, Karen Etheridge <p...@froods.org> wrote: > >> > The problem is that parent_todo is not considered when setting the OK's >> in the details in subtests. That means if a subtest ok fails, and the >> parent is in todo, the 'ok' and 'actual_ok' are both set to false. >> >> Why is this a bug? After all, it's only the top level of TAP that ever >> matters when it comes to the overall test pass/failure, and a subtest's >> pass or fail status will be adjusted accordingly for the parent TODO at the >> *parent's* level. TODOs don't need to propagate downward to all contained >> subtests. >> >> > This bug has nothing to do with TAP. '->details' is used to get a record > of all tests that have passed through the Test::Builder instance. These > details should accurately reflect if the test was a pass or fail, they do > not. By not using parent_TODO here Test::Builder is at the very least being > inconsistent with itself. When a failure occurs in a subtest where the > parent is todo Test::Builder will send the diagnostics and errors to STDOUT > instead of STDERR like it does with any TODO failure. This is the __ONLY__ > place where Test::Builder does not account for the parent_TODO on a > subtest, and in that way it is being inconsistent if not buggy. > > >> Further, if we required that TODO state must propagate down to all >> contained subtests, that means that subtests must always be aware of that >> upper state -- which rules out any use of separate processes to to create >> the subtest results, which is conceptually possible (something else, >> thinking that it is a top-level test, can provide us the TAP, and then we >> nest it as a subtest into another test -- and this should be possible both >> with TAP strings and with an event streamer). >> > > I can't control TAP from other sources, but as I said, this bug has > nothing to do with TAP. This has to do with Test::Builders internal > understanding of tests __it__ produced. Outside TAP never has, nor will it > ever effect this, they do not appear in the details structure. > > > -Chad >
