Re: [galaxy-dev] Quota History not getting purged
On Fri, Mar 14, 2014 at 1:31 PM, Alexander Kurze wrote: > Perfect! This solved my problem. > > Alex That's progress. I suspect something is going wrong in the cleanup scripts failing to update the totals - but as a workaround you could include set_user_disk_usage.py as part of the cleanup cron job? Note set_user_disk_usage.py isn't on the wiki - so I assume it isn't intended to be needed this way: https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets I've been looking at our data usage figures this week as part of a storage move - and also observed the totals can fail to drop after a user permanently deleted some large datasets & I'd run the cleanup scripts. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota History not getting purged
Perfect! This solved my problem. Alex On Fri, Mar 14, 2014 at 1:23 PM, Peter Cock wrote: > On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze > wrote: > > Hi there, > > > > We have installed a local copy of galaxy but there seems to be a strange > > behavior with deleting data sets in the history panel. A few users > complain > > that they have deleted their data sets (I can confirm that) but somehow > the > > quota limit still doesn't show the appropriate freed space. Looking at > their > > saved histories and adding up their data the quota limit should have > > changed. > > > > Any clues why this happens? I have also tried to run the clean up script > but > > this didn't do anything on their quota limits. > > > > Best, > > > > Alex > > Does this help, or make no difference? > > $ cd scripts > $ python set_user_disk_usage.py > > It will recalculate and update the user disk totals, and prints this > to the terminal too. > > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota History not getting purged
On Fri, Mar 14, 2014 at 12:28 PM, Alexander Kurze wrote: > Hi there, > > We have installed a local copy of galaxy but there seems to be a strange > behavior with deleting data sets in the history panel. A few users complain > that they have deleted their data sets (I can confirm that) but somehow the > quota limit still doesn't show the appropriate freed space. Looking at their > saved histories and adding up their data the quota limit should have > changed. > > Any clues why this happens? I have also tried to run the clean up script but > this didn't do anything on their quota limits. > > Best, > > Alex Does this help, or make no difference? $ cd scripts $ python set_user_disk_usage.py It will recalculate and update the user disk totals, and prints this to the terminal too. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Quota History not getting purged
Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota problem
Hi, Yes ! That was the problem ! I missed to put the -d value... It works like a charm now. :) Thanks a lot. Regards, Thomas. On 14/01/2014 10:00, Hans-Rudolf Hotz wrote: Hi Thomas Just double checking: what value are you using for the 'days' option? If you use the default (which is 60 days if I remember correctly), only data sets which have been 'deleted' more than 60 days ago will actually be removed. Regards, Hans-Rudolf On 01/14/2014 09:49 AM, Thomas Bellembois wrote: Hello, I am Thomas, IT engineer at the IGFL in France (http://igfl.ens-lyon.fr/). We are running our own Galaxy instance in front of an OGE cluster. Users quota never decrease when they delete items in their history. I have run the scripts in the cleanup_datasets directory but without success. I have no errors in the logs. Any idea ? Regards, Thomas ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Quota problem
Hi Thomas Just double checking: what value are you using for the 'days' option? If you use the default (which is 60 days if I remember correctly), only data sets which have been 'deleted' more than 60 days ago will actually be removed. Regards, Hans-Rudolf On 01/14/2014 09:49 AM, Thomas Bellembois wrote: Hello, I am Thomas, IT engineer at the IGFL in France (http://igfl.ens-lyon.fr/). We are running our own Galaxy instance in front of an OGE cluster. Users quota never decrease when they delete items in their history. I have run the scripts in the cleanup_datasets directory but without success. I have no errors in the logs. Any idea ? Regards, Thomas ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Quota problem
Hello, I am Thomas, IT engineer at the IGFL in France (http://igfl.ens-lyon.fr/). We are running our own Galaxy instance in front of an OGE cluster. Users quota never decrease when they delete items in their history. I have run the scripts in the cleanup_datasets directory but without success. I have no errors in the logs. Any idea ? Regards, Thomas -- Thomas Bellembois, network and system administrator ENS-Lyon/IGFL - 46, Allée d'Italie 69364 LYON CEDEX 07 FRANCE +33 4 26 73 13 67 .The IGFL is a member of the "portail-labos" project (https://portail-labos.ens-lyon.fr/site/). .IGFL IT documentation: http://itdoc.igfl.ens-lyon.fr/itdoc ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] quota
Hello Blanca, Are you still have trouble determining where you disc is being used? If so, perhaps the information in this wiki can help: http://wiki.g2.bx.psu.edu/Learn/Managing%20Datasets#Delete_vs_Delete_Permanently Please let us know if you have questions, Best, Jen Galaxy team On 3/1/12 11:43 AM, Blanca Andino wrote: My galaxy account says I am using 184 Gb of space. However, I can only account for 25.5Gb. Best Regards, Blanca ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] quota
My galaxy account says I am using 184 Gb of space. However, I can only account for 25.5Gb. Best Regards, Blanca ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
On Jan 16, 2012, at 8:31 AM, Peter Cock wrote: > On Mon, Jan 16, 2012 at 1:20 PM, graham etherington (TSL) > wrote: >> Hi Nate, >> We are getting quite large discrepancies between 'Size on disk' and >> 'Usage' numbers, so I've run the scripts/set_user_disk_usage.py script. >> When I run it, some of the 'old usage' numbers are negative numbers, e.g. >> User1 old usage: -226417886261, new usage: 139894688577 >> User2 old usage: 263698720473, new usage: 68974834617 >> User3 old usage: 397108490587, new usage: 406950046043 >> User4 old usage: -57441684074, new usage: 7536071906 >> >> This seems a little strange (how can a user have a 'minus' usage?). Is >> there an explanation for this? > > Negative numbers here are probably a sign of integer overflow > (something too large for a signed 32 bit integer would be my > guess). Possible, although I would guess that set_user_disk_usage.py was never run after the disk accounting was added (harmless, since it can be run at any time). Users who already had a bunch of data then started at '0', deleted a bunch of data, and end up with negative usage. --nate > > Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
On Mon, Jan 16, 2012 at 1:20 PM, graham etherington (TSL) wrote: > Hi Nate, > We are getting quite large discrepancies between 'Size on disk' and > 'Usage' numbers, so I've run the scripts/set_user_disk_usage.py script. > When I run it, some of the 'old usage' numbers are negative numbers, e.g. > User1 old usage: -226417886261, new usage: 139894688577 > User2 old usage: 263698720473, new usage: 68974834617 > User3 old usage: 397108490587, new usage: 406950046043 > User4 old usage: -57441684074, new usage: 7536071906 > > This seems a little strange (how can a user have a 'minus' usage?). Is > there an explanation for this? Negative numbers here are probably a sign of integer overflow (something too large for a signed 32 bit integer would be my guess). Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
Hi Nate, We are getting quite large discrepancies between 'Size on disk' and 'Usage' numbers, so I've run the scripts/set_user_disk_usage.py script. When I run it, some of the 'old usage' numbers are negative numbers, e.g. User1 old usage: -226417886261, new usage: 139894688577 User2 old usage: 263698720473, new usage: 68974834617 User3 old usage: 397108490587, new usage: 406950046043 User4 old usage: -57441684074, new usage: 7536071906 This seems a little strange (how can a user have a 'minus' usage?). Is there an explanation for this? Cheers, Graham Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601 On 06/12/2011 18:17, "Nate Coraor" wrote: >On Dec 1, 2011, at 8:37 AM, Glen Beane wrote: > >> You aren't the only one having problem in the disk-usage reported in >>the upper right. I've also had problems with this total disk usage in >>our local instance of Galaxy >> >> I've deleted all but a handful of files, and we have a cron job that >>purges the files from disk after a certain number of days. I've deleted >>everything but one history, and for the last few weeks the size of that >>history has been reported as about 200GB. I don't have any other >>histories, or any deleted but not purged datasets. However the total >>disk usage shown by Galaxy is about 1.5TB > >Hi Glen, > >There's a script to recalculate if the total becomes off, see >scripts/set_user_disk_usage.py. I've found that on the public site, the >totals are occasionally off by small amounts, meaning there must be >somewhere where data is associated with or disassociated from a user and >is not accounted for. I haven't yet found a reproducible way to make the >total incorrect, so I haven't found the bug yet. If you happen to notice >when this occurs, please let me know. Otherwise, it should be safe to >run the usage script via cron to update totals that become incorrect. > >Thanks, >--nate > >> >> >> On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: >> >>> Hi Galaxy Users, >>> I have permanently deleted numerous files. >>> My usage % has decreased, but this is NOT reflected in my Gb report >>>(upper right)- it remains the same. >>> Am I missing some phantom trash bin? >>> If not, will Galaxy recalculate my Gb usage so that I may proceed with >>>my project? >>> Thank you, >>> Mary Anne >>> >>> >>> >>> >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> >> -- >> Glen L. Beane >> Senior Software Engineer >> The Jackson Laboratory >> (207) 288-6153 >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > > >___ >Please keep all replies on the list by using "reply all" >in your mail client. To manage your subscriptions to this >and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
On Dec 6, 2011, at 2:16 PM, Mary Anne Alliegro wrote: > No, it is still reads over 120GB yet the usage is only 26% Hi Mary Anne, Sorry I didn't get back to you sooner. Your usage is currently ~75GB. The difference before may have been due to some other data in your account which was not permanently deleted. If you still find that your disk usage isn't reflecting your actual history sizes, please have a look at this page for details on how to view deleted data: http://wiki.g2.bx.psu.edu/Learn/Managing%20Datasets And let us know if this doesn't answer any remaining questions. Thanks for using Galaxy, --nate > > Mary Anne Alliegro > Marine Biological Laboratory > JOSEPHINE BAY PAUL CENTER > FOR COMPARATIVE MOLECULAR BIOLOGY AND EVOLUTION > 7 MBL Street > Woods Hole, MA 02543 > maallie...@mbl.edu > 508-289-7419 > > From: "Nate Coraor" > To: "Mary Anne Alliegro" > Cc: galaxy-...@bx.psu.edu > Sent: Tuesday, December 6, 2011 1:14:32 PM > Subject: Re: [galaxy-dev] Quota will not decrease with permanent delete > > On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: > > > Hi Galaxy Users, > > I have permanently deleted numerous files. > > My usage % has decreased, but this is NOT reflected in my Gb report (upper > > right)- it remains the same. > > Am I missing some phantom trash bin? > > If not, will Galaxy recalculate my Gb usage so that I may proceed with my > > project? > > Thank you, > > Mary Anne > > Hi Mary Anne, > > I believe your usage total is now correct, but if this is not the case, > please let us know. > > Thanks, > --nate > > > > > > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
On Dec 1, 2011, at 8:37 AM, Glen Beane wrote: > You aren't the only one having problem in the disk-usage reported in the > upper right. I've also had problems with this total disk usage in our local > instance of Galaxy > > I've deleted all but a handful of files, and we have a cron job that purges > the files from disk after a certain number of days. I've deleted everything > but one history, and for the last few weeks the size of that history has been > reported as about 200GB. I don't have any other histories, or any deleted > but not purged datasets. However the total disk usage shown by Galaxy is > about 1.5TB Hi Glen, There's a script to recalculate if the total becomes off, see scripts/set_user_disk_usage.py. I've found that on the public site, the totals are occasionally off by small amounts, meaning there must be somewhere where data is associated with or disassociated from a user and is not accounted for. I haven't yet found a reproducible way to make the total incorrect, so I haven't found the bug yet. If you happen to notice when this occurs, please let me know. Otherwise, it should be safe to run the usage script via cron to update totals that become incorrect. Thanks, --nate > > > On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: > >> Hi Galaxy Users, >> I have permanently deleted numerous files. >> My usage % has decreased, but this is NOT reflected in my Gb report (upper >> right)- it remains the same. >> Am I missing some phantom trash bin? >> If not, will Galaxy recalculate my Gb usage so that I may proceed with my >> project? >> Thank you, >> Mary Anne >> >> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > > -- > Glen L. Beane > Senior Software Engineer > The Jackson Laboratory > (207) 288-6153 > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: > Hi Galaxy Users, > I have permanently deleted numerous files. > My usage % has decreased, but this is NOT reflected in my Gb report (upper > right)- it remains the same. > Am I missing some phantom trash bin? > If not, will Galaxy recalculate my Gb usage so that I may proceed with my > project? > Thank you, > Mary Anne Hi Mary Anne, I believe your usage total is now correct, but if this is not the case, please let us know. Thanks, --nate > > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota will not decrease with permanent delete
You aren't the only one having problem in the disk-usage reported in the upper right. I've also had problems with this total disk usage in our local instance of Galaxy I've deleted all but a handful of files, and we have a cron job that purges the files from disk after a certain number of days. I've deleted everything but one history, and for the last few weeks the size of that history has been reported as about 200GB. I don't have any other histories, or any deleted but not purged datasets. However the total disk usage shown by Galaxy is about 1.5TB On Nov 30, 2011, at 2:24 PM, Mary Anne Alliegro wrote: > Hi Galaxy Users, > I have permanently deleted numerous files. > My usage % has decreased, but this is NOT reflected in my Gb report (upper > right)- it remains the same. > Am I missing some phantom trash bin? > If not, will Galaxy recalculate my Gb usage so that I may proceed with my > project? > Thank you, > Mary Anne > > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota
If you are using the unstable galaxy-central repository, universe_wsgi.ini has a new parameter: # Enable enforcement of quotas. Quotas can be set from the Admin interface. #enable_quotas = False There are new database tables so you will need to backup and run the update script for this to work. On Fri, Aug 12, 2011 at 10:56 PM, Chris Fields wrote: > I recall this being mentioned at the Galaxy Conference, and I think there are > quotas in place for the PSU servers. Maybe it's in place but not documented > yet? > > chris > > On Aug 12, 2011, at 7:49 AM, Steve Taylor wrote: > >> Hi, >> >> Are there any plans to have a disk space quota for Galaxy users that can be >> set by admins? >> >> Thanks, >> >> Steve ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Director of Bioinformatics, Channing Lab; Tel: +1 617 505 4850; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444; ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota
On Fri, Aug 12, 2011 at 1:56 PM, Chris Fields wrote: > I recall this being mentioned at the Galaxy Conference, and I > think there are quotas in place for the PSU servers. Maybe it's > in place but not documented yet? > > chris It looks like Nate committed something a week ago, https://bitbucket.org/galaxy/galaxy-central/changeset/f018b1450140 Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota
I recall this being mentioned at the Galaxy Conference, and I think there are quotas in place for the PSU servers. Maybe it's in place but not documented yet? chris On Aug 12, 2011, at 7:49 AM, Steve Taylor wrote: > Hi, > > Are there any plans to have a disk space quota for Galaxy users that can be > set by admins? > > Thanks, > > Steve ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Quota
Hi, Are there any plans to have a disk space quota for Galaxy users that can be set by admins? Thanks, Steve ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
