The primary focus of these workflow tools usually seems to be the dependency / pipeline aspect, and they appear not quite robust (at least the ones I tried) with respect to interacting with guarded HPC systems (e.g., requiring SSH multi-hop and not allowing long-running processes on login nodes). They are probably fine if we can run them directly on the login nodes, or can keep a daemon running on the login nodes. However, for the use case of setting up jobs on personal workstation, and submitting them to multiple HPC systems opportunistically, they all might require the development of some new plugins. I felt that Ansible may be a suitable framework for such plugins because it's quite reliable for various SSH scenarios and does not use a remote agent.
On Wednesday, December 27, 2017 at 6:19:55 AM UTC-6, Jimmy Tang wrote: > > Ansible probably isn't the right tool for workflows/pipelines for staging > data in and out of a cluster, you might be better of looking at something > like makeflow, luigi or airflow (there' a bunch of these pipeline/workflow > tools) We use luigi with a slurm plugin to stage data into a job and > execute whatever is needed. As others have pointed out Ansible is more > suited to setup/maintenance of a cluster. > > On Thu, Dec 21, 2017 at 6:35 PM, 'Roc' via Ansible Project < > [email protected] <javascript:>> wrote: > >> I'm exploring Ansible for managing HPC workflows. The use case is that a >> user >> prepares a set of files on a central computer (e.g., a workstation or >> laptop), >> transfers them to a bunch of HPC machines, and submits a job to their >> queuing >> systems. Once the job on any of them starts running, the other ones are >> then >> canceled. When the job finishes, the files are transfered back. >> >> It seems that Ansible has the necessary parts to transfer files, run >> commands, >> poll status, and pull info (by the remote machines), etc. I'm wondering >> if there >> are already modules to handle these types of tasks. If not, what would you >> recommend for putting up such a solution? I imagine a few new plugins and >> modules will have to be developed. This will be really useful to avoid >> the file >> syncing disaster when each of the servers has some but not all of the >> latest >> data. A lot of people roll their own impromptu solutions, but most just >> "live" >> with the pain. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ansible Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ansible-project/551c0b71-4c4d-4714-b90d-72577e5d1e98%40googlegroups.com >> >> <https://groups.google.com/d/msgid/ansible-project/551c0b71-4c4d-4714-b90d-72577e5d1e98%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > http://jcftang.github.io/ > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/25b0e8cc-b88b-418b-b0c8-4c100a17d28b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
