On Nov 27, 2012, at 9:37 AM, Peter Cock <p.j.a.c...@googlemail.com> wrote:

> On Tue, Nov 27, 2012 at 2:20 PM, Oleksandr Moskalenko <o...@hpc.ufl.edu> 
> wrote:
>> 
>> The "Right Way (TM)" I believe would be to have a universal resource request
>> selector that could be plugged into any wrapper simply by including an
>> appropriate element like say <resources proc=x pmem=y walltime=z />.
>> Those variables could be exported, so the corresponding DRMAA call
>> could be made in the dynamic runner and the data could be used in the
>> wrapper to run the underlying tool as needed.
> 
> I am not convinced about that. For a simple non-dynamic setup I
> think the resources like the number of threads should be dictated
> by the local configuration (e.g. universe_wsgi.ini) and customised
> to the local compute resources, rather than in the tool wrappers
> which must be sufficiently general to run on any Galaxy install.
> 
> In general we need dynamic negotiation between the tool (e.g. this
> tool can use as many threads as you like, suggest 8) and the local
> configuration (we want to limit this tool to just 4 threads to make
> maximum use of our cluster), and ideally the input data (e.g. this
> job will need lots of RAM and must go on the big memory queue).
> Right now the dynamic runner  which John Chilton and others
> are working on seems capable of this (although quite complex).
> 
> Regards,
> 
> Peter

The dynamic wrapper is capable of building a DRMAA call based on the external 
data and is what I am using for our local production instance. I cannot praise 
it highly enough. John Chilton has made a wonderful addition to the Galaxy. 
However, not being able to give users some manual control over the resource 
requests places the burden of figuring them out on the administrator and the 
dataset-based heuristics are often much worse then the knowledge of the person 
running the analysis. In addition, different tools use different options for 
setting thread numbers and cannot communicate realistic or even reasonable 
limits as those are based on the data from actually running the tool in 
different conditions and finding how well it scales. Dynamic negotiation is 
unfeasible at this time I think. The "simple non-dynamic setup" does not really 
work for any real-world multi-user instance anymore.

Regards,

Alex
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to