I need some feedback on some concurrency related items regarding Test-Stream.
*What you need to know:* - Test-Simple legacy supported threads, but not forking - Test::SharedFork adds for support, backcompat will be preserved - Test-Stream supports threads and forking - Test-Stream uses a single mechanism for both threading and forking (temp files for now, more later) - This email addresses some concurrency edge-cases we need to define or preserve behavior for. - I have discovered several bugs in legacy behavior *What I am looking for in feedback* - Have I missed any edge cases? - Do the Test-Stream solutions to the edge cases look sane? - Do the Test-Stream solution look like they could break anything other than existing false-passes (I am ok breaking those) - Does anyone have a solution to the thread ending after parent thread issue (The only edge-case Test-Stream does not solve, see below) *Additional discussion* - Can/Should Test-Stream wait on all child processes at the end of testing? This could break anything that uses Test::SharedFork and intentionally leaves a child running after the parent. On the other hand it helps prevent one of the edge cases that is otherwise solved by an exception. - Can/Should Test-Stream wait on all threads at the end of testing? Since it appears that perl kills any threads left running, this would only have 2 potential consequences: 1) Fix false-passes, 2) Prevent tests from ever finishing if they currently have threads with infinite loops. The benefit is that is solves the only remaining thread edge case that Test-Stream has not fixed. - The concurrency support must go out with Test-Stream, it is needed for legacy thread support and Test::SharedFork support. That said should it be a private and unpublished API for now that gets unlocked later when we like how it works, or can it be published with initial release? *The data!* I have written several scripts that demonstrate these edge cases. Each script has a stream and legacy variant, the difference is minimal in each case. I have added the output from each script to the script under __END__ for easy reference. Files can be found here: https://github.com/Test-More/test-more/tree/stream/fix_concurrency/concurrency_demos Here is my list of edge cases, as well as how legacy and stream handle them. When I mention forking in legacy it is with Test::SharedFork loaded. *These just work, when done right.* *Forking or threading inside a subtest* - Works fine in both so long as you remember to exit/wait/join in the right places *Forking or threading without subtests* - Works fine in both as long as you remember to exit/wait/join in the right places *The non-subtest Edge Cases:* *Parent ends BEFORE the child process* - In legacy, test passes as though nothing were wrong, there is an error, but the harness reports success - In Stream, initial tap passes, but a helpful message is displayed and an extra 'not ok' is sent to stdout to poison the TAP so that the harness reports failure. *Parent ends BEFORE the thread finishes (not solvable?)* - Due to the way perl threads works there is no way I can find for a thread to announce that this issue happened. - Both legacy and stream fail to alert the harness that anything is wrong, resulting in a false pass. - There is some output from perl about threads still running, but this does not cause a failure. *Subtest related edge cases:* *Forking or threading and then starting a subtest* - No issues in Test-Stream, it just works - Fails in legacy as the plan does not include the subtest, otherwise TAP output looks fine. - Test::SharedFork and thread support both have the same bug. *Failing to join a thread before the subtest completes in the parent* - In Test-Stream this throws an exception and the harness catches the failure - In Legacy this would output garbled subtest tap, but falsely report success! *Forgetting to exit the process after forking inside a subtest* - In Test-Stream this throws an exception, harness catches it, useful message is seen, no false positives. - In Legacy this outputs garbled tap, the harness catches it because some test numbers are duplicated and the plan is wrong. If there is no plan and numbers are disabled it might be a false pass. *Letting the subtest finish in the parent while a child is sending it events* - Legacy has garbled subtest TAP, may miss failures and report a false pass. - Test-Stream throws an exception with a useful message, harness notices the failure so there is no false-pass. *Forgetting to exit the child process and failing to wait on it (in subtest)* - Legacy may report false pass if numbers are disabled and there is no plan. Generally though the failure is caught by accident - Test-Stream alerts you to both issues, and causes the test to fail alerting the harness to the problem. *Here is a list of the files, with summaries:* These work fine: - fork_inside_subtest_legacy.pl - thread_inside_subtest_legacy.pl - fork_inside_subtest_stream.pl - thread_inside_subtest_stream.pl - thread_then_subtest_stream.pl - fork_then_subtest_stream.pl These are edge cases that should fail but report a pass anyway - fork_inside_subtest_noexit_legacy.pl - fork_inside_subtest_nowait_legacy.pl - parent_ends_before_fork_legacy.pl - thread_inside_subtest_nojoin_legacy.pl - parent_ends_before_thread_legacy.pl (Might not be fixable) - parent_ends_before_thread_stream.pl (Might not be fixable) These should fail, but only due so because of other factors - fork_inside_subtest_nowait-or-exit_legacy.pl - fork_then_subtest_legacy.pl - thread_then_subtest_legacy.pl These should fail, and do so by intent: - fork_inside_subtest_noexit_stream.pl - fork_inside_subtest_nowait-or-exit_stream.pl - fork_inside_subtest_nowait_stream.pl - parent_ends_before_fork_stream.pl - thread_inside_subtest_nojoin_stream.pl
