On Mon, Dec 21, 2015 at 5:10 PM, Karen Etheridge <p...@froods.org> wrote:
> > The problem is that parent_todo is not considered when setting the OK's > in the details in subtests. That means if a subtest ok fails, and the > parent is in todo, the 'ok' and 'actual_ok' are both set to false. > > Why is this a bug? After all, it's only the top level of TAP that ever > matters when it comes to the overall test pass/failure, and a subtest's > pass or fail status will be adjusted accordingly for the parent TODO at the > *parent's* level. TODOs don't need to propagate downward to all contained > subtests. > > This bug has nothing to do with TAP. '->details' is used to get a record of all tests that have passed through the Test::Builder instance. These details should accurately reflect if the test was a pass or fail, they do not. By not using parent_TODO here Test::Builder is at the very least being inconsistent with itself. When a failure occurs in a subtest where the parent is todo Test::Builder will send the diagnostics and errors to STDOUT instead of STDERR like it does with any TODO failure. This is the __ONLY__ place where Test::Builder does not account for the parent_TODO on a subtest, and in that way it is being inconsistent if not buggy. > Further, if we required that TODO state must propagate down to all > contained subtests, that means that subtests must always be aware of that > upper state -- which rules out any use of separate processes to to create > the subtest results, which is conceptually possible (something else, > thinking that it is a top-level test, can provide us the TAP, and then we > nest it as a subtest into another test -- and this should be possible both > with TAP strings and with an event streamer). > I can't control TAP from other sources, but as I said, this bug has nothing to do with TAP. This has to do with Test::Builders internal understanding of tests __it__ produced. Outside TAP never has, nor will it ever effect this, they do not appear in the details structure. -Chad
