Title: [282674] trunk/Tools
- Revision
- 282674
- Author
- [email protected]
- Date
- 2021-09-17 11:15:41 -0700 (Fri, 17 Sep 2021)
Log Message
Speed up run-jsc-stress-tests with parallel processing
https://bugs.webkit.org/show_bug.cgi?id=230251
Patch by Geza Lore <[email protected]> on 2021-09-17
Reviewed by Yusuke Suzuki.
Around 2/3 of the time spent in the initial serial processing phase of
run-jsc-stress-tests is simply writing out the many test runner
scripts to disk, which is trivially parallelizable. The change in this
patch forks sub-processes to emit the test runner scripts which makes
the serial startup phase overall about 2.5x faster on my test machine
(47 sec -> 19 sec).
* Scripts/run-jsc-stress-tests:
Modified Paths
Diff
Modified: trunk/Tools/ChangeLog (282673 => 282674)
--- trunk/Tools/ChangeLog 2021-09-17 18:07:48 UTC (rev 282673)
+++ trunk/Tools/ChangeLog 2021-09-17 18:15:41 UTC (rev 282674)
@@ -1,3 +1,19 @@
+2021-09-17 Geza Lore <[email protected]>
+
+ Speed up run-jsc-stress-tests with parallel processing
+ https://bugs.webkit.org/show_bug.cgi?id=230251
+
+ Reviewed by Yusuke Suzuki.
+
+ Around 2/3 of the time spent in the initial serial processing phase of
+ run-jsc-stress-tests is simply writing out the many test runner
+ scripts to disk, which is trivially parallelizable. The change in this
+ patch forks sub-processes to emit the test runner scripts which makes
+ the serial startup phase overall about 2.5x faster on my test machine
+ (47 sec -> 19 sec).
+
+ * Scripts/run-jsc-stress-tests:
+
2021-09-17 Lauro Moura <[email protected]>
REGRESSION(r275267) [GLIB] API test /webkit/WebKitWebsiteData/configuration is failing
Modified: trunk/Tools/Scripts/run-jsc-stress-tests (282673 => 282674)
--- trunk/Tools/Scripts/run-jsc-stress-tests 2021-09-17 18:07:48 UTC (rev 282673)
+++ trunk/Tools/Scripts/run-jsc-stress-tests 2021-09-17 18:15:41 UTC (rev 282674)
@@ -1997,6 +1997,53 @@
end
end
+def parallelEach(array, &block)
+ # Short of using 'require "parallel"' we use a simple statically
+ # partitioned multiprocess dispatch for processing fixed chunks of the
+ # given array in parallel. We use Process rather than Thread to
+ # parallelise CPU load as well as IO (due to the GIL).
+
+ # Some platforms (notably Windows) do not support Process.fork, so work
+ # serially on these.
+ nWorkers = Process.respond_to?(:fork) ? $numChildProcesses : 1
+
+ # Chunk size is rounded up
+ chunkSize = (array.size + (nWorkers - 1)) / nWorkers
+
+ # If chunk size is too small, work serially
+ if chunkSize <= 1
+ nWorkers = 1
+ chunkSize = array.size
+ end
+
+ childPIDs = []
+
+ # Chunks 1 to nWorkers-1 run in the worker processes
+ for i in 1...nWorkers do
+ pid = Process.fork
+ if pid.nil?
+ # Worker process. Process chunk i.
+ array.slice(i*chunkSize, chunkSize).each(&block)
+ Process.exit!(true)
+ else
+ childPIDs << pid
+ end
+ end
+
+ # Main process. Process chunk 0.
+ array.slice(0, chunkSize).each(&block)
+
+ # Wait for workers
+ for pid in childPIDs do
+ _, status = Process.waitpid2(pid)
+ raise "Child process still running" unless status.exited?
+ if status.exitstatus != 0
+ STDERR.puts "Child process failed with status: #{status.exitstatus}"
+ exit(status.exitstatus)
+ end
+ end
+end
+
def prepareTestRunner(remoteIndex=0)
raise if $bundle
@@ -2019,10 +2066,12 @@
File.unlink($runnerDir + filename)
}
- $runlist.each {
- | plan |
+ # Write test scripts in parallel as this is both an expensive and a
+ # highly IO intensive operation, but each script is independent and
+ # the operation is pure other than writing the unique run script.
+ parallelEach($runlist) do | plan |
plan.writeRunScript($runnerDir + "test_script_#{plan.index}")
- }
+ end
case $testRunnerType
when :make
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes
