Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Oleksandr Moskalenko
On Nov 27, 2012, at 9:37 AM, Peter Cock wrote: > On Tue, Nov 27, 2012 at 2:20 PM, Oleksandr Moskalenko > wrote: >> >> The "Right Way (TM)" I believe would be to have a universal resource request >> selector that could be plugged into any wrapper simply by including an >> appropriate element l

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 2:20 PM, Oleksandr Moskalenko wrote: > > The "Right Way (TM)" I believe would be to have a universal resource request > selector that could be plugged into any wrapper simply by including an > appropriate element like say . > Those variables could be exported, so the corres

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Oleksandr Moskalenko
On Nov 27, 2012, at 6:06 AM, Andreas Kuntzagk wrote: > Hi Peter, > > thanks for your replies. > > On 27.11.2012 11:44, Peter Cock wrote: >> On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk >> wrote: >>> Dear Peter, >>> As the author of several tool wrappers, I've been asking for a Gal

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Hi Peter, thanks for your replies. On 27.11.2012 11:44, Peter Cock wrote: On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk wrote: Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for e

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 10:44 AM, Andreas Kuntzagk wrote: > Hi Alex, > >> I am not sure if you can call these "surprises". > > Well at least it surprised me :-) > Didn't want to sound to negative. > >> Some tools (which I highly appreciate) of Peter have been "parallelised" >> to get the job done

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 10:38 AM, Andreas Kuntzagk wrote: > Dear Peter, > >> As the author of several tool wrappers, I've been asking for a Galaxy >> wide mechanism for Galaxy to tell the tool how many threads it can >> use, for example via an environment variable. The value could then >> be set w

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
galaxy-dev-boun...@lists.bx.psu.edu] Namens Andreas Kuntzagk Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called "lastz" and ran in parallel and consu

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Dear Peter, As the author of several tool wrappers, I've been asking for a Galaxy wide mechanism for Galaxy to tell the tool how many threads it can use, for example via an environment variable. The value could then be set with a general default, per runner default, or even per tool using the ex

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Peter Cock
On Tue, Nov 27, 2012 at 8:58 AM, Andreas Kuntzagk wrote: > Hi, > > the four processes I saw where all called "lastz" and ran in parallel and > consumed 100% of a core each. > My guess is that the lastz_wrapper.py is responsible for this. > Looking at it I see a some code regarding queuing and in t

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Bossers, Alex
x.psu.edu] Namens Andreas Kuntzagk Verzonden: dinsdag 27 november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: Re: [galaxy-dev] multithreaded tools Hi, the four processes I saw where all called "lastz" and ran in parallel and consumed 100% of a core each. My gu

Re: [galaxy-dev] multithreaded tools

2012-11-27 Thread Andreas Kuntzagk
Hi, the four processes I saw where all called "lastz" and ran in parallel and consumed 100% of a core each. My guess is that the lastz_wrapper.py is responsible for this. Looking at it I see a some code regarding queuing and in the very beginning this line: WORKERS = 4 and further one the cla

Re: [galaxy-dev] multithreaded tools

2012-11-26 Thread Bob Harris
Howdy, Andreas, The four processes started for a galaxy lastz job must involve post- processing the lastz output through some other shell tool. Lastz by itself doesn't support multiple threads or processes. Bob H On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote: Hi, I'm wandering ho

Re: [galaxy-dev] multithreaded tools

2012-11-26 Thread Bossers, Alex
-dev-boun...@lists.bx.psu.edu] Namens Andreas Kuntzagk Verzonden: maandag 26 november 2012 9:58 Aan: galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] multithreaded tools Hi, I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed th

[galaxy-dev] multithreaded tools

2012-11-26 Thread Andreas Kuntzagk
Hi, I'm wandering how galaxy supports tools that are multithreaded or multi-process. When working with lastz I noticed that it starts 4 parallel processes. Is that always so? Can this be adjusted? What other tools also are multi-process? regards, Andreas -- Andreas Kuntzagk SystemAdministrato