Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-05 Thread David Kovalic
Enis, FYI, Unfortunately I am traveling out of the country today and further investigation will need to wait until I return. I will set a note to follow-up on this when I get back and let you know more details so we can see just what is happening in more detail. Expect to hear back from me in a

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-04 Thread Enis Afgan
> > > Now we have what may be a CloudMan question: our working cluster has a > head node and 4 workers. The head node is loaded up but the workers are > idle. I would have thought jobs should be pushing out to the workers but we > don't see any load on these machines. > > So are the jobs queued jus

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-03 Thread David Kovalic
Peter, Thanks for the great information. I see where to tune the "split_size" variable and also the READMEs :) I'll do so more sleuthing, let the job run and observe. So far it is ~3hr after job launch and still no load on the workers. I think from looking at the /mnt/galaxy/tmp/job_working_dire

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-03 Thread Peter Cock
On Tue, May 3, 2016 at 9:23 PM, David Kovalic wrote: > Peter, > > We made the modification to the config file, restarted galaxy and things > seem to be working from the galaxy end. We see sub-job directories being > created in /mnt/galaxy/tmp/job_working_directory. We think all of the > required j

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-03 Thread David Kovalic
Peter, We made the modification to the config file, restarted galaxy and things seem to be working from the galaxy end. We see sub-job directories being created in /mnt/galaxy/tmp/job_working_directory. We think all of the required job chunks have been created (i.e. total sequences/1000 sub-job di

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-03 Thread David Kovalic
Peter, Thanks, I didn't see that, I was reading the paper and searching online. Appreciate the help, we'll give it a go! David On Tue, May 3, 2016 at 2:16 PM Peter Cock wrote: > Hi David, > > The NCBI BLAST+ wrappers have a tag setup, > which becomes active if you have use_tasked_jobs = Tru

Re: [galaxy-dev] Parallelism (job splitting) for ncbi_blast_plus running through CloudMan

2016-05-03 Thread Peter Cock
Hi David, The NCBI BLAST+ wrappers have a tag setup, which becomes active if you have use_tasked_jobs = True in your config/galaxy.ini file (aka universe_wsgi.ini). Specifically, the wrappers use this: This is hard coded to break up the query FASTA file into batches of 1000 sequences (e.g. a

Re: [galaxy-dev] Parallelism using metadata

2015-07-06 Thread Peter Cock
BOF wiki page here, will add time and place once settled... https://wiki.galaxyproject.org/Events/GCC2015/BoFs/DataSplittingAndParallelism On Mon, Jul 6, 2015 at 6:22 PM, Peter Cock wrote: > Hello all, > > Marco, Jon and I are going to try to get together during GCC2015 > for a BOF (a "Birds of

Re: [galaxy-dev] Parallelism using metadata

2015-07-06 Thread Peter Cock
Hello all, Marco, Jon and I are going to try to get together during GCC2015 for a BOF (a "Birds of a Feather" informal meeting) to talk about Galaxy parallelisation (and collections). We've not yet picked a time, but details should be on the wiki shortly... https://wiki.galaxyproject.org/Events/G

Re: [galaxy-dev] Parallelism using metadata

2015-03-03 Thread Marco Albuquerque
Hi John, Thanks for your reply. I think for the time being, I will simply create a tool that creates an interval file, and the parallelize on this interval file. Though I agree, this would be a useful feature to include but I don't think I am anywhere near ready to start dabbling in galaxy's core

Re: [galaxy-dev] Parallelism using metadata

2015-03-02 Thread John Chilton
Hey Marco, Thanks for the e-mail. This is an awesome idea, but I am worried it is very hard to do this well in Galaxy. If you create symbolic links to the original file - then Galaxy might delete the original file and the derived files would all break without warning. Galaxy does have this separat

Re: [galaxy-dev] parallelism

2015-02-05 Thread Daniel Blankenberg
Hi Roberto, It looks like you are probably still using the default sqlite database. You’ll want to update to using e.g. postgres when exploring these more database intensive functions. See e.g.: https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Switching_to_a_database_ser