Re: [galaxy-dev] Scaling tools: data available? (Peter van Heusden)

[Peter van Heusden]

Thanks Damion

We've got the collectl based metric collection stuff working and that dumps
data in the Galaxy DB (which you can pull out using the API).

Thanks,
Peter

On 17 March 2016 at 10:45, Dooley, Damion  wrote:

> If you are looking for a simple python program to log cpu% and memory of
> another executable, then analyze.py might do.  Its in
> https://github.com/Public-Health-Bioinformatics/kipper/tree/master/RDP-test
> -case along with a few scripts that show examples of its use.
>
> Damion
> >>
> >> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
> >>
> >> Hi there
> >>
> >> We're researching the resource (CPU time and memory) requirements for
> >>RNA
> >> STAR at the moment. Specifically, we'd like to build up a database of
> >>input
> >> size to resource usage so that we can use this to feed a dynamic
> >> destination mapper (currently our STAR configuration uses a thumbsuck
> >>for
> >> RAM and CPU requirements that is not optimal but works in many cases).
> >>
> >> Has anyone collected these metrics before? And more generally, is anyone
> >> collecting metrics for a range of bioinformatics tools? A database like
> >> this could be very useful!
> >>
> >> Thanks!
> >> Peter
> >>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling tools: data available? (Peter van Heusden)

[Dooley, Damion]

If you are looking for a simple python program to log cpu% and memory of
another executable, then analyze.py might do.  Its in
https://github.com/Public-Health-Bioinformatics/kipper/tree/master/RDP-test
-case along with a few scripts that show examples of its use.

Damion
>>
>> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
>>
>> Hi there
>>
>> We're researching the resource (CPU time and memory) requirements for
>>RNA
>> STAR at the moment. Specifically, we'd like to build up a database of
>>input
>> size to resource usage so that we can use this to feed a dynamic
>> destination mapper (currently our STAR configuration uses a thumbsuck
>>for
>> RAM and CPU requirements that is not optimal but works in many cases).
>>
>> Has anyone collected these metrics before? And more generally, is anyone
>> collecting metrics for a range of bioinformatics tools? A database like
>> this could be very useful!
>>
>> Thanks!
>> Peter
>>

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling tools: data available?

[Eric Rasche]

Hi Peter,

The vision I had was a cron script admins could install which would send
thr past day/week's collectl/memory/parameter/dataset size logs to GRT.

Again, the GRT side stalled when I hit issues with collectl locally. If you
have time/energy to work on that we should chat :-) Otherwise I plan to
find time for it before GCC, minus the collectl stuff, just capturing
runtimes as a start.

Ciao,
Eric
14. mar. 2016 1.09 p.m. skrev "Peter van Heusden" :

> Hi Eric
>
> I'm having a look at the GRT. We're collecting metrics on our local Galaxy
> server, I'm just trying to understand how you envision these metrics
> getting into the GRT?
>
> Peter
>
> On 22 February 2016 at 07:31, Eric Rasche  wrote:
>
>> Hi peter,
>>
>> I've been working on this in the past but got caught up in other
>> projects. There's work for a server-side component here (maybe 50% done),
>> the client side stuff could all be done through collectl but I was having
>> issues actually collecting collectl data that had put the project on hold
>> temporarily.
>>
>> https://github.com/erasche/galactic-radio-telescope
>>
>> I'd be very, very interested in finishing this project up.
>>
>> Cheers,
>> Eric
>>
>>
>> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
>>
>> Hi there
>>
>> We're researching the resource (CPU time and memory) requirements for RNA
>> STAR at the moment. Specifically, we'd like to build up a database of input
>> size to resource usage so that we can use this to feed a dynamic
>> destination mapper (currently our STAR configuration uses a thumbsuck for
>> RAM and CPU requirements that is not optimal but works in many cases).
>>
>> Has anyone collected these metrics before? And more generally, is anyone
>> collecting metrics for a range of bioinformatics tools? A database like
>> this could be very useful!
>>
>> Thanks!
>> Peter
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>>
>> --
>> --
>> Eric Rasche
>> Programmer II
>>
>> Center for Phage Technology
>> Rm 312A, BioBio
>> Texas A University
>> College Station, TX 77843
>> 404-692-2048
>> e...@tamu.edu
>> Not responding quickly enough?
>> 
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling tools: data available?

[Peter van Heusden]

Hi Eric

I'm having a look at the GRT. We're collecting metrics on our local Galaxy
server, I'm just trying to understand how you envision these metrics
getting into the GRT?

Peter

On 22 February 2016 at 07:31, Eric Rasche  wrote:

> Hi peter,
>
> I've been working on this in the past but got caught up in other projects.
> There's work for a server-side component here (maybe 50% done), the client
> side stuff could all be done through collectl but I was having issues
> actually collecting collectl data that had put the project on hold
> temporarily.
>
> https://github.com/erasche/galactic-radio-telescope
>
> I'd be very, very interested in finishing this project up.
>
> Cheers,
> Eric
>
>
> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
>
> Hi there
>
> We're researching the resource (CPU time and memory) requirements for RNA
> STAR at the moment. Specifically, we'd like to build up a database of input
> size to resource usage so that we can use this to feed a dynamic
> destination mapper (currently our STAR configuration uses a thumbsuck for
> RAM and CPU requirements that is not optimal but works in many cases).
>
> Has anyone collected these metrics before? And more generally, is anyone
> collecting metrics for a range of bioinformatics tools? A database like
> this could be very useful!
>
> Thanks!
> Peter
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
> --
> --
> Eric Rasche
> Programmer II
>
> Center for Phage Technology
> Rm 312A, BioBio
> Texas A University
> College Station, TX 77843
> 404-692-2048
> e...@tamu.edu
> Not responding quickly enough?
> 
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Scaling tools: data available?

[Peter van Heusden]

Hi there

We're researching the resource (CPU time and memory) requirements for RNA
STAR at the moment. Specifically, we'd like to build up a database of input
size to resource usage so that we can use this to feed a dynamic
destination mapper (currently our STAR configuration uses a thumbsuck for
RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone
collecting metrics for a range of bioinformatics tools? A database like
this could be very useful!

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/