Hi Alex,

I am not sure if you can call these "surprises".

Well at least it surprised me :-)
Didn't want to sound to negative.

Some tools (which I highly appreciate) of Peter have been "parallelised" to get 
the job done more
quickly. I earlier mentioned the ncbi blast+ wrappers but there the tool by 
itself handles the
multithreading. Other tools I am aware that use a python script/wrapper to 
chunk up the initial
query and rejoin later are tools like signalp, TMHMM and such. Usually it also 
involves some
parsing of output to data that galaxy can subsequently handle. In the latter 
examples its done
using python scripts, but for some of our custom tools we did it in perl, some 
using bash
parallel, or using R. I wouldn't have a solution to getting to know this 
without going through
the initial wrappers...


While I can read Python and bash fine it becomes more complicated with perl and R. Don't know if I could easily spot from the code what the number of threads is.
So maybe somebody could setup a list of these tools?

regards, Andreas.


Alex


-----Oorspronkelijk bericht----- Van: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens Andreas Kuntzagk Verzonden: 
dinsdag 27
november 2012 9:58 Aan: Bob Harris CC: galaxy-dev@lists.bx.psu.edu Onderwerp: 
Re: [galaxy-dev]
multithreaded tools

Hi,

the four processes I saw where all called "lastz" and ran in parallel and 
consumed 100% of a core
each. My guess is that the lastz_wrapper.py is responsible for this. Looking at 
it I see a some
code regarding queuing and in the very beginning this line:

WORKERS = 4

and further one the class BaseQueue which starts "threads". BTW. there seems to 
be no way to
adjust this number other than editing the source file - bad.

And this get's me wondering if there are other such surprises hidden in galaxy.

regards, Andreas

On 26.11.2012 15:55, Bob Harris wrote:
Howdy, Andreas,

The four processes started for a galaxy lastz job must involve post-processing 
the lastz output
through some other shell tool.  Lastz by itself doesn't support multiple 
threads or processes.

Bob H


On Nov 26, 2012, at 3:58 AM, Andreas Kuntzagk wrote:

Hi,

I'm wandering how galaxy supports tools that are multithreaded or 
multi-process. When working
with lastz I noticed that it starts 4 parallel processes. Is that always so? 
Can this be
adjusted? What other tools also are multi-process?

regards, Andreas

-- Andreas Kuntzagk

SystemAdministrator

Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for 
Molecular
Medicine Robert-Roessle-Str. 10, 13125 Berlin, Germany

http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich
___________________________________________________________ Please keep all 
replies on the
list by using "reply all" in your mail client.  To manage your subscriptions to 
this and
other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/


-- Andreas Kuntzagk

SystemAdministrator

Berlin Institute for Medical Systems Biology at the Max-Delbrueck-Center for 
Molecular Medicine
Robert-Roessle-Str. 10, 13125 Berlin, Germany

http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich
___________________________________________________________ Please keep all 
replies on the list
by using "reply all" in your mail client.  To manage your subscriptions to this 
and other Galaxy
lists, please use the interface at:

http://lists.bx.psu.edu/





--
Andreas Kuntzagk

SystemAdministrator

Berlin Institute for Medical Systems Biology at the
Max-Delbrueck-Center for Molecular Medicine
Robert-Roessle-Str. 10, 13125 Berlin, Germany

http://www.mdc-berlin.de/en/bimsb/BIMSB_groups/Dieterich
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Reply via email to