Title: [286128] trunk/Tools
- Revision
- 286128
- Author
- [email protected]
- Date
- 2021-11-23 00:57:03 -0800 (Tue, 23 Nov 2021)
Log Message
Fix buildbot command timeout issue for remote tests
https://bugs.webkit.org/show_bug.cgi?id=233309
Reviewed by Adrian Perez de Castro.
Occasionally, buildbots that execute the JSC tests on remote devices
would stop producing any output and eventually get killed due to a timeout.
The issue is that run-jsc-stress-tests would start an ssh command
(overwhelmingly likely, the command to copy over the payload), then the remote
board would stop responding.
Introduce a timeout option to forEachRemote and bound the waiting time.
Could also do this for each command individually (and this would make it
straightforward to GC zombie processes) but seems way more fragile than
doing it at the thread level.
* Scripts/run-jsc-stress-tests:
Modified Paths
Diff
Modified: trunk/Tools/ChangeLog (286127 => 286128)
--- trunk/Tools/ChangeLog 2021-11-23 08:54:06 UTC (rev 286127)
+++ trunk/Tools/ChangeLog 2021-11-23 08:57:03 UTC (rev 286128)
@@ -1,3 +1,24 @@
+2021-11-23 Angelos Oikonomopoulos <[email protected]>
+
+ Fix buildbot command timeout issue for remote tests
+ https://bugs.webkit.org/show_bug.cgi?id=233309
+
+ Reviewed by Adrian Perez de Castro.
+
+ Occasionally, buildbots that execute the JSC tests on remote devices
+ would stop producing any output and eventually get killed due to a timeout.
+
+ The issue is that run-jsc-stress-tests would start an ssh command
+ (overwhelmingly likely, the command to copy over the payload), then the remote
+ board would stop responding.
+
+ Introduce a timeout option to forEachRemote and bound the waiting time.
+ Could also do this for each command individually (and this would make it
+ straightforward to GC zombie processes) but seems way more fragile than
+ doing it at the thread level.
+
+ * Scripts/run-jsc-stress-tests:
+
2021-11-22 Wenson Hsieh <[email protected]>
Remove old concurrent display list logic that's no longer necessary
Modified: trunk/Tools/Scripts/run-jsc-stress-tests (286127 => 286128)
--- trunk/Tools/Scripts/run-jsc-stress-tests 2021-11-23 08:54:06 UTC (rev 286127)
+++ trunk/Tools/Scripts/run-jsc-stress-tests 2021-11-23 08:57:03 UTC (rev 286128)
@@ -81,6 +81,7 @@
ITERATION_LIMITS = OpenStruct.new(:infraIterationsFloor => 3,
:iterationsCeiling => 10)
+REMOTE_TIMEOUT = 120
begin
require 'shellwords'
rescue Exception => e
@@ -2601,12 +2602,40 @@
}
}
+ etime = nil
+ if options.has_key?(:timeout)
+ etime = Time.now + options[:timeout]
+ end
liveRemotes = []
threads.each_index {
| index |
thread = threads[index]
begin
- thread.join
+ if options.has_key?(:timeout)
+ if etime.nil?
+ # If a timeout has been requested and etime is nil,
+ # that means the timeout has expired and we shouldn't
+ # wait at all.
+ timeout = 0
+ else
+ timeout = etime - Time.now
+ if timeout < 0
+ timeout = 0
+ end
+ end
+ if thread.join(timeout).nil?
+ if $verbosity > 0
+ $stderr.puts("Timeout joining thread for remote #{remoteHosts[index]}")
+ end
+ # Timeout expired, so we can't block waiting for
+ # any other threads. Either they're done or
+ # they've also timed out.
+ etime = nil
+ raise CommandExecutionFailed
+ end
+ else
+ thread.join # No timeout requested, just block.
+ end
liveRemotes << remoteHosts[index]
rescue CommandExecutionFailed
if options[:dropOnFailure]
@@ -2694,7 +2723,7 @@
end
def unpackBundleGnuParallel(remoteHosts)
- forEachRemote(remoteHosts, :dropOnFailure => true) {
+ forEachRemote(remoteHosts, :dropOnFailure => true, :timeout => REMOTE_TIMEOUT) {
| _, remoteHost |
mysys(["ssh", "-o", "NoHostAuthenticationForLocalhost=yes"] +
(remoteHost.identity_file_path ? ["-i", remoteHost.identity_file_path] : []) +
@@ -2863,7 +2892,7 @@
# list. Otherwise, if it comes back online in the middle of an
# iteration, we'll try to run test jobs on it, possibly using
# an unrelated bundle from a previous run.
- remoteHosts = forEachRemote(remoteHosts, {:dropOnFailure => true}) {
+ remoteHosts = forEachRemote(remoteHosts, {:dropOnFailure => true, :timeout => REMOTE_TIMEOUT}) {
| _, remoteHost |
getRemoteDirectoryIfNeeded(remoteHost)
copyBundleToRemote(remoteHost)
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes
