We are looking at integrating some tools into Galaxy for a domain
where many of the existing programs are Windows only. My desire is to
write tool files and have the certain jobs run remotely on a Windows
host (our Galaxy servers are Linux). It wasn't clear how to do this with
the existing job runners since they all seem to require a shared file
system and absolute file paths that are the same on the Galaxy server as
they are on the remote execution hosts.
I developed a new job runner that allows for the remote execution
of jobs that also rewrites paths and stages files as part of the job
execution. There is a server application that needs to be running on the
remote machine (*nix or Windows). This application is likely easier to
setup/configure than SGE/PBS/etc... because it is a simple Python paste
application, with just a few configuration options. There may be valid
use cases for this approach for non-Windows applications as well - if
you would like a lighter weight server component on the remote system,
if shared file systems are impossible/impractical, etc....
The path rewrites are not going to cover every possible way an
absolute path can sneak into the execution, so this is only useful for a
subset of tools. But it does handle normal file inputs and outputs,
config files, and scripts in the tool definition directory. I welcome
input/advice on this approach or the implementation. If there is any way
I can improve the code base to the point where it can be included in
galaxy-central I would be more than happy to make the desired changes.
If not, it is out there for people to use if they have a similar use case.
The server application as well as a patch to add the runner/client
to a galaxy-central installation can be found at:
Thanks for your time,
University of Minnesota Supercomputing Institute
galaxy-dev mailing list