Hi Dan, I suspect if you dig in a bit, you'll probably be looking at a pretty major architectural change to Ansible's TQM to avoid the subprocess overhead, as we currently fork host workers before the connection. Making a ParallelSSH-based connection plugin should be pretty straightforward, but I'm not sure you'd see a lot of benefit under Ansible's current process model (and would probably actually see a perf regression for lightly-loaded scenarios).
I've got selfish reasons for wanting to see the TQM process model abstracted (*cough* native Windows support *cough*) but it'd be a pretty big meatball IMO. If you really want to pursue it, that's a thing that's big enough to write a proposal for <https://github.com/ansible/proposals/>. -Matt On Tuesday, June 28, 2016 at 9:12:36 AM UTC-7, Dan wrote: > > Bumpity bump :) > > Is there any interest in having asynchronous parallel SSH connections in > Ansible? > > The use case is tasks in parallel over SSH on a number of servers which is > currently comparatively slow and very resource intensive, cpu usage and > system load wise. > > On Monday, June 6, 2016 at 10:58:35 AM UTC+1, Dan wrote: >> >> Hello dev list, >> >> Many thanks for your efforts on ansible, very useful tool. >> >> Have a new feature I'd like to work on that would like to float by you >> for comments and if it sounds like a good feature to implement before >> submitting a PR. >> >> In short, would like to add the ability to use the ParallelSSH client >> library <https://github.com/pkittenis/parallel-ssh> to run SSH commands >> via ansible. The library uses paramiko and gevent to provide an >> asynchronous parallel SSH client python library. >> >> This would greatly benefit ansible when running multiple commands in >> parallel. The current parallel implementation uses multiprocessing and >> increases system load on the host running ansible linearly as # of threads >> increases. >> >> Benefits of using the ParallelSSH library: >> >> * Asynchronous parallel SSH commands >> * No system load induced on the host running it, regardless of # of >> parallel connections >> * Persistent connections by default, one connection per host >> * Stdout/stderr and exit code retrieval built in >> * Configurable parallelism level >> * Native proxying/tunneling support, also asynchronous. No ProxyCommand, >> no subprocess per tunnel >> * Native SFTP support >> * Pure python library, no cmd line tools used >> * No openssh or other ssh cmd line tool dependencies >> * Posix systems and windows compatible >> >> Effect on ansible: >> * Uses gevent's monkey patching to make paramiko asynchronous. Monkey >> patching of subprocess in particular will also affect ansible's use of >> subprocess, though it should be compatible. >> * gevent library would be a new requirement for ansible >> >> >> Is this of interest? If so, where do you think it should go, a new >> connection type perhaps or overriding existing paramiko implementation? I >> would think overriding paramiko implementation to use the library would >> make most sense. As the library itself uses paramiko it is already >> compatible. >> >> As a bonus if use of paramiko is overridden, most of the code in >> paramiko_ssh.py >> <https://github.com/pkittenis/ansible/blob/devel/lib/ansible/plugins/connection/paramiko_ssh.py> >> >> can go as the client implementation lives entirely in the library. >> >> Would argue that the subprocess use of the ssh binary in ssh.py should be >> overridden as well, at least for parallel commands, but given backwards >> compatibility requirements doubt that would be feasible. >> >> Comments appreciated, thanks. >> >> - Dan >> >> -- You received this message because you are subscribed to the Google Groups "Ansible Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
