Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line: WORKERS = 4 and further one the

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Bossers, Alex
Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 8:58 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value could then be set with a general default, per runner default, or even per tool using the

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Andreas Kuntzagk Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called lastz and ran in parallel and consumed 100% of a core each. My guess

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 10:44 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi Alex, I am not sure if you can call these surprises. Well at least it surprised me :-) Didn't want to sound to negative. Some tools (which I highly appreciate) of Peter have been parallelised to

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Hi Peter, thanks for your replies. On 27.11.2012 11:44, Peter Cock wrote: On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Oleksandr Moskalenko
On Nov 27, 2012, at 6:06 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Hi Peter, thanks for your replies. On 27.11.2012 11:44, Peter Cock wrote: On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: Dear Peter, As the author of several

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 2:20 PM, Oleksandr Moskalenko o...@hpc.ufl.edu wrote: The Right Way (TM) I believe would be to have a universal resource request selector that could be plugged into any wrapper simply by including an appropriate element like say resources proc=x pmem=y walltime=z /.

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Oleksandr Moskalenko
On Nov 27, 2012, at 9:37 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Nov 27, 2012 at 2:20 PM, Oleksandr Moskalenko o...@hpc.ufl.edu wrote: The Right Way (TM) I believe would be to have a universal resource request selector that could be plugged into any wrapper simply by

Re: [galaxy-dev] multithreaded tools

2012-11-26 Thread Bossers, Alex
Andreas, Yes this is possible. You can also have a look at the ncbi blast+ tools written by Peter. The same is true. Usually the tool wrappers (xml) have an option preconfigured how many threads can be used... You can adjust these directly in the xml OR as we did we added the option to be

Re: [galaxy-dev] multithreaded tools

2012-11-26 Thread Bob Harris
Howdy, Andreas, The four processes started for a galaxy lastz job must involve post- processing the lastz output through some other shell tool. Lastz by itself doesn't support multiple threads or processes. Bob H On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote: Hi, I'm wandering