Branch: refs/heads/smoke-me/khw-harness Home: https://github.com/Perl/perl5 Commit: 1c5a64cf361ac893e240c2b6c441e72a959bf9c6 https://github.com/Perl/perl5/commit/1c5a64cf361ac893e240c2b6c441e72a959bf9c6 Author: Karl Williamson <k...@cpan.org> Date: 2020-04-12 (Sun, 12 Apr 2020)
Changed paths: M cpan/Test-Harness/lib/App/Prove.pm M cpan/Test-Harness/lib/App/Prove/State.pm M cpan/Test-Harness/lib/App/Prove/State/Result.pm M cpan/Test-Harness/lib/App/Prove/State/Result/Test.pm M cpan/Test-Harness/lib/TAP/Base.pm M cpan/Test-Harness/lib/TAP/Formatter/Base.pm M cpan/Test-Harness/lib/TAP/Formatter/Color.pm M cpan/Test-Harness/lib/TAP/Formatter/Console.pm M cpan/Test-Harness/lib/TAP/Formatter/Console/ParallelSession.pm M cpan/Test-Harness/lib/TAP/Formatter/Console/Session.pm M cpan/Test-Harness/lib/TAP/Formatter/File.pm M cpan/Test-Harness/lib/TAP/Formatter/File/Session.pm M cpan/Test-Harness/lib/TAP/Formatter/Session.pm M cpan/Test-Harness/lib/TAP/Harness.pm M cpan/Test-Harness/lib/TAP/Harness/Env.pm M cpan/Test-Harness/lib/TAP/Object.pm M cpan/Test-Harness/lib/TAP/Parser.pm M cpan/Test-Harness/lib/TAP/Parser/Aggregator.pm M cpan/Test-Harness/lib/TAP/Parser/Grammar.pm M cpan/Test-Harness/lib/TAP/Parser/Iterator.pm M cpan/Test-Harness/lib/TAP/Parser/Iterator/Array.pm M cpan/Test-Harness/lib/TAP/Parser/Iterator/Process.pm M cpan/Test-Harness/lib/TAP/Parser/Iterator/Stream.pm M cpan/Test-Harness/lib/TAP/Parser/IteratorFactory.pm M cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm M cpan/Test-Harness/lib/TAP/Parser/Result.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Bailout.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Comment.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Plan.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Pragma.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Test.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Unknown.pm M cpan/Test-Harness/lib/TAP/Parser/Result/Version.pm M cpan/Test-Harness/lib/TAP/Parser/Result/YAML.pm M cpan/Test-Harness/lib/TAP/Parser/ResultFactory.pm M cpan/Test-Harness/lib/TAP/Parser/Scheduler.pm M cpan/Test-Harness/lib/TAP/Parser/Scheduler/Job.pm M cpan/Test-Harness/lib/TAP/Parser/Scheduler/Spinner.pm M cpan/Test-Harness/lib/TAP/Parser/Source.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Executable.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/File.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Handle.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Perl.pm M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/RawTAP.pm M cpan/Test-Harness/lib/TAP/Parser/YAMLish/Reader.pm M cpan/Test-Harness/lib/TAP/Parser/YAMLish/Writer.pm M cpan/Test-Harness/lib/Test/Harness.pm Log Message: ----------- TAP::Harness: Move timer initialization Prior to this commit, the timers for counting elapsed time and CPU usage were begun when a job's first output appears. This yields inaccurate results. These results are saved in t/test_state for future runs so that they can start the longest-running tests first, which leads to using the available cores more efficiently. (If you start a long running test after everything else is nearly done, you have to wait for it to finish before the suite as a whole is; if you start the long ones first, and the shortest last, you don't have to wait very long for any stragglers to complete.) Inaccurate results here lead to this situation, which we were often seeing in the podcheck.t test. The worst case is if there is heavy computation at the beginning of the test being run. podcheck, for example, examines all the pods in the directory structure to find which links to other pods do or do not have corresponding anchors. Output doesn't happen until the analysis is complete. On my system, this takes over 30 seconds, but prior to this commit, what was noted was just the time required to do the output, about 200 milliseconds. The result was that podcheck was viewed as being one of the shortest tests run, so was started late in the process, and generally held up the completion of it. This commit by itself doesn't improve the test completion very much, because, test tests are run a whole directory at a time, and the directory podcheck is in, for example, is run last. The next commit addresses that. Commit: 7a4bc22907055ec090421ffef0fa853d999efa04 https://github.com/Perl/perl5/commit/7a4bc22907055ec090421ffef0fa853d999efa04 Author: Karl Williamson <k...@cpan.org> Date: 2020-04-12 (Sun, 12 Apr 2020) Changed paths: M t/harness Log Message: ----------- XXX env name: t/harness: Add option for faster test suite execution This commit adds an environment variable, XXX, which if set to non-zero increases the parallelism in the execution of the test suite, speeding it up on systems with multiple cores. Normally, there are two main test sections, one for core and the second for non-core tests, and the testing of the non-core one doesn't begin until the first is complete. Within each section, there are a number of test categories, like 're' for regular expressions, and 'JSON::PP' for the pure perl implementation of JSON. Within each category, there are various single .t test files. Some categories can have those be tested in parallel; some require them to be done in a particular order, say because an earlier .t does setup for subsequent ones. We already have this capability. Completion of all the tests in a category is not needed before those of another category can be started. This is how it already works. However, the core section categories are ordered so that they begin in a logical order for someone trying to get perl to work. First to start are the basic sanity tests, then by roughly decreasing order of widespread use in perl programs in the wild, with the final two categories, porting and perf, being mainly of use to perl5 porters. These two categories aren't started until all the tests in the earlier categories are started. We have some long running tests in those two categories, and generally they delay the start of the entire second section. If those long running tests could be started sooner, shorter tests in the first section could be run in parallel with them, increasing the average CPU utilization, and the second section could begin (and hence end) earlier, shortening the total elapsed execution time of the entire suite. The second section has some very long running tests. JSON-PP is one of them. If it could run in parallel with tests from the first section, that would also speed up the completion of the suite. The environment variable added by this commit does both things. The basic sanity test categories in the first section continue to be started before anything else. But then all other tests are run in decreasing order of elapsed time they take to run, removing the boundaries between some categories, and between the two sections. The gain from this increases as the number of jobs run in parallel does; slower high core platforms have the highest increase. On the old dromedary with 24 cores, the gain is 20%, almost 2 minutes. On my more modern box with 12 cores, it is 8%. Compare: https://github.com/Perl/perl5/compare/b92d8089883c...7a4bc2290705