jenkins-bot has submitted this change and it was merged.
Change subject: Add scap.ssh module
......................................................................
Add scap.ssh module
This patch provides a replacement for DSH that is implemented in Python. It
works by having a generator that forks subprocesses of OpenSSH and collates
their output and exit statuses.
To keep things manageable, this patch merely introduces the SSH module.
A subsequent patch will replace invocations of dsh with scap.ssh.cluster_ssh
and scap.ssh.cluster_run.
Basic strategy: on each iteration, make a non-blocking check for terminated
children via os.waitpid w/the os.WNOHANG flag. Then drain any output buffers
that are ready to be read, using non-block IO. Then reap terminated children,
if there were any.
What's the point, if we're simply replacing Popen(dsh) with Popen(ssh)? Because
it allows us to monitor the progress of the command as it runs, and because we
know the outcome (stdout + stderr + exit status) for each host without having to
reconstruct it from DSH's collated output.
Change-Id: I6a52a93dc0083979d903becb9c768c6a24d82e03
---
A scap/ssh.py
1 file changed, 58 insertions(+), 0 deletions(-)
Approvals:
BryanDavis: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scap/ssh.py b/scap/ssh.py
new file mode 100644
index 0000000..9ae9df9
--- /dev/null
+++ b/scap/ssh.py
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+"""
+ scap.ssh
+ ~~~~~~~~
+ This module provides functions for running commands on remote hosts
+ via SSH.
+
+"""
+import os
+import select
+import subprocess
+
+
+SSH = ('/usr/bin/ssh', '-oBatchMode=yes', '-oSetupTimeout=10')
+
+
+def cluster_ssh(hosts, command, limit=80):
+ """Run a command via SSH on multiple hosts concurrently."""
+ hosts = set(hosts)
+ procs = {}
+ fds = {}
+ poll = select.epoll()
+ while hosts or procs:
+ if hosts and len(procs) < limit:
+ host = hosts.pop()
+ ssh_command = SSH + (host, command)
+ proc = subprocess.Popen(ssh_command, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT, preexec_fn=os.setsid)
+ procs[proc.pid] = (proc, host)
+ poll.register(proc.stdout, select.EPOLLIN)
+ else:
+ pid, status = os.waitpid(-1, os.WNOHANG)
+ for fd, event in poll.poll(0.01):
+ fds[fd] = fds.get(fd, '') + os.read(fd, 1048576)
+ if pid:
+ status = -(status & 255) or (status >> 8)
+ proc, host = procs.pop(pid)
+ poll.unregister(proc.stdout)
+ output = fds.pop(proc.stdout.fileno(), '')
+ yield host, status, output
+ poll.close()
+
+
+def cluster_run(hosts, command, max_fails=0):
+ """Run a command via SSH on multiple hosts concurrently. Wait until
+ all spawned processes complete, then return a tuple of (ok, failed)
+ mappings."""
+ max_fails = round(max_fails)
+ failed = {}
+ ok = {}
+ for host, status, output in cluster_ssh(hosts, command):
+ if status == 0:
+ ok[host] = status, output
+ else:
+ failed[host] = status, output
+ if len(failed) > max_fails:
+ raise RuntimeError(command, failed)
+ return ok, failed
--
To view, visit https://gerrit.wikimedia.org/r/113660
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6a52a93dc0083979d903becb9c768c6a24d82e03
Gerrit-PatchSet: 6
Gerrit-Project: mediawiki/tools/scap
Gerrit-Branch: master
Gerrit-Owner: Ori.livneh <[email protected]>
Gerrit-Reviewer: BryanDavis <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
