On Tue, Nov 27, 2012 at 8:58 AM, Andreas Kuntzagk
<andreas.kuntz...@mdc-berlin.de> wrote:
> Hi,
>
> the four processes I saw where all called "lastz" and ran in parallel and
> consumed 100% of a core each.
> My guess is that the lastz_wrapper.py is responsible for this.
> Looking at it I see a some code regarding queuing and in the very beginning
> this line:
>
> WORKERS = 4
>
> and further one the class BaseQueue which starts "threads". BTW. there seems
> to be no way to adjust this number other than editing the source file - bad.
>
> And this get's me wondering if there are other such surprises hidden in
> galaxy.
>
> regards, Andreas

As the author of several tool wrappers, I've been asking for a Galaxy
wide mechanism for Galaxy to tell the tool how many threads it can
use, for example via an environment variable. The value could then
be set with a general default, per runner default, or even per tool
using the existing runner configuration under [galaxy:tool_runners]
in universe_wsgi.ini

See: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-March/009037.html
and: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html

In your example, and others like the BWA and BLAST+ wrappers where
the tool XML is hard coded to 8 threads, you would probably want to
use a custom runner in universe_wsgi.ini setting the cluster submission
to request that many slots/CPUs.

For our local cluster, I modify the BLAST+ wrapper XML to use 4 threads,
and have something like this in my universe_wsgi.ini file:

[galaxy:tool_runners]
ncbi_blastp_wrapper  = drmaa://-V -pe smp 4/
ncbi_blastn_wrapper  = drmaa://-V -pe smp 4/
ncbi_blastx_wrapper  = drmaa://-V -pe smp 4/
ncbi_tblastn_wrapper = drmaa://-V -pe smp 4/
ncbi_tblastx_wrapper = drmaa://-V -pe smp 4/

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to