Hi Ben,
The job running code is in lib/galaxy/jobs/. Galaxy jobs get a
wrapper which include the start/finish methods, that's in
__init__.py. handler.py is what dispatches jobs out to various runner
plugins, finds new jobs to run, and generally controls the operation.
runners/*.py are the
You've been extremely helpful, I appreciate it.
So we went ahead and decided that we need this feature. We're planning to
have a lot of people running huge pipelines, ones that work best on one
node, and there's no reason to do all this writing to any shared file
system when it works best on one
File system performance varies wildly between storage architectures.
There are storage server setups that can easily scale to orders of
magnitude beyond the compute that backs usegalaxy.org - suffice to say
we are currently bound by the number of cores we have available and
not by IO/network
Hey Ben,
Thanks for the e-mail. I did not promise anything was coming soon, I
only said people were working on parts of it. It is not a feature yet
unfortunately - multiple people including myself are thinking about
various parts of this problem though.
I would like to respond, but I am trying
Hi John, thanks for the reply.
Yes, I mean Galaxy's default behavior of keeping all the data on all nodes
of our condor cluster. So for instance if I run a job, then the output of
that job is copied to every node in the cluster. Is this not the normal
behavior?
On Tue, Dec 17, 2013 at 9:42 AM,
How do you have it set up on the main public galaxy install? I imagine that
people run enough big jobs that there there is enormous use of your shared
file system. How did you scale that to so many nodes without bogging down
the file system with large dataset transfers?
It seems that for now the
This does mostly make sense, and is very illuminating. I appreciate all the
help and I'm sorry that I'm so new to this.
I'm not sure I fully understand though. Do you mean that I could have a
main Galaxy install setup for the 200 nodes, for general purpose use with a
shared file system, and a
