Re: [galaxy-dev] Data Libraries are slow

[Dannon Baker]

Hey Iyad,

This is an area under active development.  You can track the progress of it
on this card here:  https://trello.com/c/nwYQNFPK/56-data-library-evolution

The new data libraries are *much* faster, and if you'd like to enable them
for your local site.  You should be able to see this in current versions of
galaxy under Shared Data -> Data Libraries Beta.

-Dannon


On Thu, May 29, 2014 at 8:26 AM, Kandalaft, Iyad 
wrote:

>  Hello everyone
>
>
>
> I have a lot of sequencing data that I wanted to load into one Galaxy data
> library.  I enabled loading the data by filesystem path and successfully
> “linked to the files” rather than copying them into galaxy.  Unfortunately,
> as I add more folders and data files, the library takes longer and longer
> to load.  At this point, the apache proxy just times out before galaxy
> returns any data.  Should I have more data libraries and fewer files within
> them?  Is there a reason for not implementing paging in the data libraries?
>
>
>
> The web server hosting galaxy shows only 80% loads
>
> 30473 galaxy20   0 1070m 443m 5160 S 81.6  1.4   9:59.89 python
>
>
>
> The database server is at 13% load (kind of low)
>
> 20890 mysql 20   0 3762m 551m 9104 S 13.4  0.9 796:16.69
> /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql
> --plugin-dir=/usr/lib64/mysql/plugin --user=mysql
> --log-bin=/var/log/mysql/bin.log
> --slow-query-log-file=/var/log/mysql/slow-queries.log
>
>
>
>
>
> Regards,
>
> Iyad Kandalaft
>
>
>
> Bioinformatics Application Developer
>
> Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
>
> KW Neatby Bldg | éd. KW Neatby
>
> 960 Carling Ave| 960, avenue Carling
>
> Ottawa, ON | Ottawa (ON) K1A 0C6
>
> E-mail Address / Adresse courriel: iyad.kandal...@agr.gc.ca
>
> Telephone | Téléphone 613- 759-1228
>
> Facsimile | Télécopieur 613-759-1701
>
> Government of Canada | Gouvernement du Canada
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Data Libraries are slow

[Kandalaft, Iyad]

Hello everyone

I have a lot of sequencing data that I wanted to load into one Galaxy data 
library.  I enabled loading the data by filesystem path and successfully 
"linked to the files" rather than copying them into galaxy.  Unfortunately, as 
I add more folders and data files, the library takes longer and longer to load. 
 At this point, the apache proxy just times out before galaxy returns any data. 
 Should I have more data libraries and fewer files within them?  Is there a 
reason for not implementing paging in the data libraries?

The web server hosting galaxy shows only 80% loads
30473 galaxy20   0 1070m 443m 5160 S 81.6  1.4   9:59.89 python

The database server is at 13% load (kind of low)
20890 mysql 20   0 3762m 551m 9104 S 13.4  0.9 796:16.69 /usr/sbin/mysqld 
--basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin 
--user=mysql --log-bin=/var/log/mysql/bin.log 
--slow-query-log-file=/var/log/mysql/slow-queries.log


Regards,
Iyad Kandalaft

Bioinformatics Application Developer
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
KW Neatby Bldg | éd. KW Neatby
960 Carling Ave| 960, avenue Carling
Ottawa, ON | Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel: 
iyad.kandal...@agr.gc.ca
Telephone | Téléphone 613- 759-1228
Facsimile | Télécopieur 613-759-1701
Government of Canada | Gouvernement du Canada

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/