I need some feedback on some concurrency related items regarding
Test-Stream.
*What you need to know:*
- Test-Simple legacy supported threads, but not forking
- Test::SharedFork adds for support, backcompat will be preserved
- Test-Stream supports threads and forking
- Test-Stream uses a single mechanism for both threading and forking
(temp files for now, more later)
- This email addresses some concurrency edge-cases we need to define or
preserve behavior for.
- I have discovered several bugs in legacy behavior
*What I am looking for in feedback*
- Have I missed any edge cases?
- Do the Test-Stream solutions to the edge cases look sane?
- Do the Test-Stream solution look like they could break anything other
than existing false-passes (I am ok breaking those)
- Does anyone have a solution to the thread ending after parent thread
issue (The only edge-case Test-Stream does not solve, see below)
*Additional discussion*
- Can/Should Test-Stream wait on all child processes at the end of
testing? This could break anything that uses Test::SharedFork and
intentionally leaves a child running after the parent. On the other hand it
helps prevent one of the edge cases that is otherwise solved by an
exception.
- Can/Should Test-Stream wait on all threads at the end of testing?
Since it appears that perl kills any threads left running, this would only
have 2 potential consequences: 1) Fix false-passes, 2) Prevent tests from
ever finishing if they currently have threads with infinite loops. The
benefit is that is solves the only remaining thread edge case that
Test-Stream has not fixed.
- The concurrency support must go out with Test-Stream, it is needed for
legacy thread support and Test::SharedFork support. That said should it be
a private and unpublished API for now that gets unlocked later when we like
how it works, or can it be published with initial release?
*The data!*
I have written several scripts that demonstrate these edge cases. Each
script has a stream and legacy variant, the difference is minimal in each
case. I have added the output from each script to the script under __END__
for easy reference.
Files can be found here:
https://github.com/Test-More/test-more/tree/stream/fix_concurrency/concurrency_demos
Here is my list of edge cases, as well as how legacy and stream handle
them. When I mention forking in legacy it is with Test::SharedFork loaded.
*These just work, when done right.*
*Forking or threading inside a subtest*
- Works fine in both so long as you remember to exit/wait/join in the
right places
*Forking or threading without subtests*
- Works fine in both as long as you remember to exit/wait/join in the
right places
*The non-subtest Edge Cases:*
*Parent ends BEFORE the child process*
- In legacy, test passes as though nothing were wrong, there is an
error, but the harness reports success
- In Stream, initial tap passes, but a helpful message is displayed and
an extra 'not ok' is sent to stdout to poison the TAP so that the harness
reports failure.
*Parent ends BEFORE the thread finishes (not solvable?)*
- Due to the way perl threads works there is no way I can find for a
thread to announce that this issue happened.
- Both legacy and stream fail to alert the harness that anything is
wrong, resulting in a false pass.
- There is some output from perl about threads still running, but this
does not cause a failure.
*Subtest related edge cases:*
*Forking or threading and then starting a subtest*
- No issues in Test-Stream, it just works
- Fails in legacy as the plan does not include the subtest, otherwise
TAP output looks fine.
- Test::SharedFork and thread support both have the same bug.
*Failing to join a thread before the subtest completes in the parent*
- In Test-Stream this throws an exception and the harness catches the
failure
- In Legacy this would output garbled subtest tap, but falsely report
success!
*Forgetting to exit the process after forking inside a subtest*
- In Test-Stream this throws an exception, harness catches it, useful
message is seen, no false positives.
- In Legacy this outputs garbled tap, the harness catches it because
some test numbers are duplicated and the plan is wrong. If there is no plan
and numbers are disabled it might be a false pass.
*Letting the subtest finish in the parent while a child is sending it
events*
- Legacy has garbled subtest TAP, may miss failures and report a false
pass.
- Test-Stream throws an exception with a useful message, harness notices
the failure so there is no false-pass.
*Forgetting to exit the child process and failing to wait on it (in
subtest)*
- Legacy may report false pass if numbers are disabled and there is no
plan. Generally though the failure is caught by accident
- Test-Stream alerts you to both issues,
