Interesting!
But maybe it is not the manager that is the bottleneck, but which process is the one that takes the inputs from the webpage (or in the API) and prepares all the files that are produced by the workflows?

I guess it is the MAIN server?
THAT is the one that is the bottleneck I think then...

Are there any plans to make that process parallel?

What I am facing, is that I have 100 FASTQ pairs or so, for a single flowcell, I can start the analysis of that set from the UI, but it will just crank through them and takes about 2-4 minutues for each pair to be processed, so with 100 pairs or so, you are looking 3-4 hours of an hourglass before control is given back to the user...

It would be ideal if we would farm out the parsing of the web input to the cluster, so the creation of the histories and datasets is no longer the bottleneck....

How do others deal with large sequencing projects on this list?
Are there any groups that routinely do 100-200 pairs of FASTQ files from within galaxy?

I would like to hear their experience...

Thanks

Thon

On Feb 13, 2013, at 01:44 PM, Nate Coraor <n...@bx.psu.edu> wrote:

On Feb 13, 2013, at 4:12 PM, Anthonius deBoer wrote:

> Would this problem go away if I only used ONE webserver in my proxy setting and not use a balancer?
> I may not need it and if it means I could not use the toolshed effectively, it may be worth it not to use balancing..

No, because your manager/handlers will still need to be restarted to pick up the new tools. The only way you can have it work without restarting is to have everything in a single process.

> What i really need is the manager being balanced, since submitting a workflow with 100's of BAM files can take hours just to get started...ANy change we can run multiple managers?

I'm surprised the manager is the holdup here since all it does is assign a handler. Regardless, in the development branch (which will be the next stable/release branch), the manager has been killed, and handlers are assigned directly by the web processes.

--nate

>
> Thanks
>
> On Feb 13, 2013, at 07:59 AM, Nate Coraor <n...@bx.psu.edu> wrote:
>
>> On Feb 7, 2013, at 2:43 PM, Anthonius deBoer wrote:
>>
>> > That's very unfortunate...
>> > I have a ton of tools and I guess now I have to create a package for them in a local toolshed to update them in a running galaxy server?
>> >
>> > In any case...The toolshed installation also does not work for me...I still have to restart galaxy, even after using the toolshed approach to install a tool...It either does not show up at all or give a bunch of errors, about not being able to find the tool...
>> >
>> > Is this also related to the fact I have two webservers and am behind a proxy server as well?
>>
>> Hi Thon,
>>
>> Essentially yes, there is no way for one web process to communicate with others that it has installed a tool. We'd like to allow for this sort of notification via a message queue, but we don't have a proper message queue in Galaxy right now.
>>
>> --nate
>>
>> >
>> > Thon
>> >
>> > On Feb 07, 2013, at 05:29 AM, Dannon Baker <dannonba...@me.com> wrote:
>> >
>> >> Unfortunately not, and with the migration of tools to the toolshed installation mechanism I don't imagine this will be addressed (at least by the team) anytime soon. If you wanted you could probably write a script that would reload a specified tool in each of the separate web processes, or just implement a complete rolling restart of your web processes to avoid service disruption while still loading the tool updates.
>> >>
>> >> -Dannon
>> >>
>> >>
>> >> On Feb 6, 2013, at 8:40 PM, Anthonius deBoer <thondeb...@me.com> wrote:
>> >>
>> >> > I am indeed using multiple web processes and I guess I am talking about the "old" admine tool reloader...
>> >> > Is there any other way to do this for your own tools that you just manually place in tools etc.?
>> >> >
>> >> > Thon
>> >> >
>> >> > On Feb 05, 2013, at 06:22 PM, Dannon Baker <dannonba...@me.com> wrote:
>> >> >
>> >> >> Are you using multiple web processes, and are you referring to the old admin tool reloader or the toolshed reloading interface?
>> >> >>
>> >> >> -Dannon
>> >> >>
>> >> >> On Feb 5, 2013, at 9:13 PM, Anthonius deBoer <thondeb...@me.com> wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > I find that reloading a tool's configuration file does not really work.
>> >> >> > First, you have to click the reload buttow twice to actually have it update the VERSION number (so it does read something)...
>> >> >> > But when I try to run my tool, the old bug is still there...
>> >> >> >
>> >> >> > I am using proxy server so something may still be cached, but I have to restart my server for it actually to pick up the changes...
>> >> >> >
>> >> >> > Any ideas?
>> >> >> >
>> >> >> > Thon
>> >> >> > ___________________________________________________________
>> >> >> > Please keep all replies on the list by using "reply all"
>> >> >> > in your mail client. To manage your subscriptions to this
>> >> >> > and other Galaxy lists, please use the interface at:
>> >> >> >
>> >> >> > http://lists.bx.psu.edu/
>> >> >>
>> >>
>> > ___________________________________________________________
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client. To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >
>> > http://lists.bx.psu.edu/
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to