Following up on some recent threads that have referenced my dynamic
job runner configuration work, Nate and I have discussed these issues
and I have created a new pull request based on those discussions and I
am confident these changes will be accepted soon.
Things are basically as I outlined them in my previous description:
http://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg03010.html
except for where to place the rules has changed. Now instead of
placing them in lib/galaxy/jobs/rules.py you will need to create a
file (or multiple files for them) in your lib/galaxy/jobs/rules the
directory.
My previous e-mail was a technical description of how it worked, I
think maybe that is why it didn't generate the excitement I had hoped
:). I think instead describing some concrete use cases might be
better. So here are six cool things you can do with dynamic job
runners.
1) Change maximum walltime based on job parameters or file sizes.
2) Implement wild card like configuration of job runners instead of
configuring one tool at a time.
3) Create queues with different priorities, and then give higher
priorities to people giving demos (or directors or testers etc...).
4) Utilize environment variables to determine job runner configurations.
5) Limit a particular tool's use to only white-listed users.
6) Tie into Galaxy's job history tables to throttle those problem
users clogging up your Galaxy instance.
To do any of these you will need pull in the changes from bitbucket,
add dynamic to the start_job_runners configuration option in
universe_wsgi.ini, and create a file such as
lib/galaxy/jobs/rules/200_runners.py for your rules.
Below I describe how to do these, though I haven't actually tested the
code snippets so they should be considered just an outline of the
idea, your mileage may vary.
1) Change maximum walltime based on job parameters or file sizes.
Lets say you want to change the max walltime of the BlastN based on
the size of the input query. First you would add the line
ncbi_blastn_wrapper=dynamic:///python to universe_wsgi.ini. Next in
200_runners.py you would add a function such as following:
import os
def ncbi_blastn_wrapper(job):
inp_data = dict( [ ( da.name, da.dataset ) for da in
job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for
da in job.input_library_datasets ] )
query_file = inp_data[ "query" ].file_name
query_size = os.path.getsize( query_file )
if query_size > 1024 * 1024:
return 'pbs:-l walltime=24:00:00/'
else:
return 'pbs:-l walltime=12:00:00/'
2) Implement wild card like configuration of job runners instead of
configuring one tool at a time.
Lets say you have a coworker called J. Johnson ummm wait no Jim J. and
he maintains a tool suite for a fictitious metagenomics application
called fathur. Assume also that this fathur suite has dozens of tools
clogging up your configuration file because they need to all use
pbs:-l procs=8/ instead of the default pbs:/. To configure all
the fathur tools at once, in the [app:main] of universe_wsgi.ini would
would change default_cluster_job_runner from pbs:/ to
dynamic:///python/default_runner and then add the following function
to 200_runners.py.
def default_runner(tool_id):
if tool_id.startswith('fathur_'):
return 'pbs:-l procs=8/'
else:
return 'pbs:/'
3) Create queues with different priorities, and then give higher
priorities to people giving demos.
Lets say the users defined by the admin_users configuration property
in universe_wsgi.ini are the ones that give demos and do testing and
so you want to increase their priority for all jobs, and lets say to
do this you have created queues gx_normal and gx_important in your
queue manager with differing priorities. You could then take the
default_runner concept from the previous example and do something like
this:
def default_runner(app, user_email):
admin_users = app.config.get( "admin_users", "" ).split( "," )
if user_email in admin_users:
return 'pbs:///gx_important//'
else:
return 'pbs:///gx_normal//'
You could define the list of users right in this file instead of
pulling it in from admin_users and then apply this concept to give
higher priority to director or testers or paying users. Alternatively,
you could give lower priority to external users, people you just don't
like, etc
4) Utilize environment variables to determine job runner configurations.
Lets say you want cufflinks to always use as many cores as are
available, but in your testing environment you only have 4 cores
available whereas in production you have 16. Lets also say you have
the environment variable MAX_CORES set and this will be different on
each machine. You would then update universe_wsgi.ini to have
cufflinks use the dynamic job config (cufflinks=dynamic:///python) and
then add the following to 200_runners.py
import os
def cufflinks():
return 'pbs:-l procs=%s/' % os.environ['MAX_CORES']
(Warning you would need to update the cuffli
