Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
The cleanup scripts enforce a sort of "lifetime" for the datasets. The first time they're run, they may mark a dataset as deleted and also reset the update time and you'll have to wait N days for the next stage of the lifetime. The next time they're run, or if a dataset has already been marked as deleted, the actual file removal happens and purged is set to true (if it wasn't already). You can manually pass in '-d 0' to force removal of datasets recently marked as deleted. The purge scripts do not check 'allow_user_dataset_purge', of course. On Tue, Mar 18, 2014 at 11:50 AM, Carl Eberhard wrote: > I believe it's a (BAD) silent failure mode in the server code. > > If I understand correctly, the purge request isn't coughing an error when > it gets to the 'allow_user_dataset_purge' check and instead is silently > marking (or re-marking) the datasets as deleted. > > I would rather it fail with a 403 error if purge is explicitly requested. > > That said, it of course would be better to remove the purge operation > based on the configuration then to show an error after we've found you > can't do the operation. The same holds true for the 'permanently remove > this dataset' link in deleted datasets. > > I'll see if I can find out the answer to your question on the cleanup > scripts. > > > On Tue, Mar 18, 2014 at 10:49 AM, Peter Cock wrote: > >> On Tue, Mar 18, 2014 at 2:14 PM, Carl Eberhard >> wrote: >> > Thanks, Ravi & Peter >> > >> > I've added a card to get the allow_user_dataset_purge options into the >> > client and to better show the viable options to the user: >> > https://trello.com/c/RCPZ9zMF >> >> Thanks Carl - so this was a user interface bug, showing the user >> non-functional permanent delete (purge) options. That's clearer now. >> >> In this situation can the user just 'delete', and wait N days for >> the cleanup scripts to actually purge the files and free the space? >> (It seems N=10 in scripts/cleanup/purge_*.sh at least, elsewhere >> like the underlying Python script the default looks like N=60). >> >> Regards, >> >> Peter >> > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
I believe it's a (BAD) silent failure mode in the server code. If I understand correctly, the purge request isn't coughing an error when it gets to the 'allow_user_dataset_purge' check and instead is silently marking (or re-marking) the datasets as deleted. I would rather it fail with a 403 error if purge is explicitly requested. That said, it of course would be better to remove the purge operation based on the configuration then to show an error after we've found you can't do the operation. The same holds true for the 'permanently remove this dataset' link in deleted datasets. I'll see if I can find out the answer to your question on the cleanup scripts. On Tue, Mar 18, 2014 at 10:49 AM, Peter Cock wrote: > On Tue, Mar 18, 2014 at 2:14 PM, Carl Eberhard > wrote: > > Thanks, Ravi & Peter > > > > I've added a card to get the allow_user_dataset_purge options into the > > client and to better show the viable options to the user: > > https://trello.com/c/RCPZ9zMF > > Thanks Carl - so this was a user interface bug, showing the user > non-functional permanent delete (purge) options. That's clearer now. > > In this situation can the user just 'delete', and wait N days for > the cleanup scripts to actually purge the files and free the space? > (It seems N=10 in scripts/cleanup/purge_*.sh at least, elsewhere > like the underlying Python script the default looks like N=60). > > Regards, > > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Tue, Mar 18, 2014 at 2:14 PM, Carl Eberhard wrote: > Thanks, Ravi & Peter > > I've added a card to get the allow_user_dataset_purge options into the > client and to better show the viable options to the user: > https://trello.com/c/RCPZ9zMF Thanks Carl - so this was a user interface bug, showing the user non-functional permanent delete (purge) options. That's clearer now. In this situation can the user just 'delete', and wait N days for the cleanup scripts to actually purge the files and free the space? (It seems N=10 in scripts/cleanup/purge_*.sh at least, elsewhere like the underlying Python script the default looks like N=60). Regards, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
Thanks, Ravi & Peter I've added a card to get the allow_user_dataset_purge options into the client and to better show the viable options to the user: https://trello.com/c/RCPZ9zMF On Fri, Mar 14, 2014 at 11:10 AM, Peter Cock wrote: > On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock > wrote: > > On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi wrote: > >> I do not think so. Several individual datasets have been deleted > (clicked > >> the upper-right X on the history item box) but no History has been > >> permanently deleted. > >> > >> Is there any indication in the database if target dataset or datasets > were > >> marked for permanent deletion? In the dataset table, I see fields > >> "deleted", "purged", and "purgable", but nothing that says permanently > >> deleted. > > > > I would welcome clarification from the Galaxy Team, here and > > on the wiki page which might benefit from a flow diagram? > > > > > https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets > > > > My assumption is using "permanently delete" in the user interface > > marks an entry as "purgable", and then it will be moved to "purged" > > (and the associated file on disk deleted) by the cleanup scripts - > > but I'm a bit hazy on this any why it takes a while for a user's > > usage figures to change. > > Hmm. Right now I've unable (via the web interface) to permanently > delete a history - it stays stuck as "deleted", and thus (presumably) > won't get purged by the clean up scripts. > > I've tried: > > 1. Load problem history > 2. Rename the history "DIE DIE" to avoid confusion > 3. Top right menu, "Delete permanently" > 4. Prompted "Really delete the current history permanently? This > cannot be undone", OK > 5. Told "History deleted, a new history is active" > 6. Top right menu, "Saved Histories" > 7. Click "Advanced Search", status "all" > 8. Observe "DIE DIE" history is only "deleted" (while other older > histories are "deleted permanently") (BAD) > 9. Run the cleanup scripts, > > $ sh scripts/cleanup_datasets/delete_userless_histories.sh > $ sh scripts/cleanup_datasets/purge_histories.sh > $ sh scripts/cleanup_datasets/purge_libraries.sh > $ sh scripts/cleanup_datasets/purge_folders.sh > $ sh scripts/cleanup_datasets/purge_datasets.sh > > 10. Reload the saved history list, no change. > 11. Using the drop down menu, select "Delete Permanently" > 12. Prompted "History contents will be removed from disk, this cannot > be undone. Continue", OK > 13. No change to history status (BAD) > 14. Tick the check-box, and use the "Delete Permanently" button at the > bottom of the page > 15. Prompted "History contents will be removed from disk, this cannot > be undone. Continue", OK > 16. No change to history status (BAD) > 17. Run the cleanup scripts, no change. > > Note that in my universe_wsgi.ini I have not (yet) set: > allow_user_dataset_purge = True > > If this setting is important, then the interface seems confused - > and if quotas are enforced, very frustrating :( > > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock wrote: > On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi wrote: >> I do not think so. Several individual datasets have been deleted (clicked >> the upper-right X on the history item box) but no History has been >> permanently deleted. >> >> Is there any indication in the database if target dataset or datasets were >> marked for permanent deletion? In the dataset table, I see fields >> "deleted", "purged", and "purgable", but nothing that says permanently >> deleted. > > I would welcome clarification from the Galaxy Team, here and > on the wiki page which might benefit from a flow diagram? > > https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets > > My assumption is using "permanently delete" in the user interface > marks an entry as "purgable", and then it will be moved to "purged" > (and the associated file on disk deleted) by the cleanup scripts - > but I'm a bit hazy on this any why it takes a while for a user's > usage figures to change. Hmm. Right now I've unable (via the web interface) to permanently delete a history - it stays stuck as "deleted", and thus (presumably) won't get purged by the clean up scripts. I've tried: 1. Load problem history 2. Rename the history "DIE DIE" to avoid confusion 3. Top right menu, "Delete permanently" 4. Prompted "Really delete the current history permanently? This cannot be undone", OK 5. Told "History deleted, a new history is active" 6. Top right menu, "Saved Histories" 7. Click "Advanced Search", status "all" 8. Observe "DIE DIE" history is only "deleted" (while other older histories are "deleted permanently") (BAD) 9. Run the cleanup scripts, $ sh scripts/cleanup_datasets/delete_userless_histories.sh $ sh scripts/cleanup_datasets/purge_histories.sh $ sh scripts/cleanup_datasets/purge_libraries.sh $ sh scripts/cleanup_datasets/purge_folders.sh $ sh scripts/cleanup_datasets/purge_datasets.sh 10. Reload the saved history list, no change. 11. Using the drop down menu, select "Delete Permanently" 12. Prompted "History contents will be removed from disk, this cannot be undone. Continue", OK 13. No change to history status (BAD) 14. Tick the check-box, and use the "Delete Permanently" button at the bottom of the page 15. Prompted "History contents will be removed from disk, this cannot be undone. Continue", OK 16. No change to history status (BAD) 17. Run the cleanup scripts, no change. Note that in my universe_wsgi.ini I have not (yet) set: allow_user_dataset_purge = True If this setting is important, then the interface seems confused - and if quotas are enforced, very frustrating :( Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi wrote: > I do not think so. Several individual datasets have been deleted (clicked > the upper-right X on the history item box) but no History has been > permanently deleted. > > Is there any indication in the database if target dataset or datasets were > marked for permanent deletion? In the dataset table, I see fields > "deleted", "purged", and "purgable", but nothing that says permanently > deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using "permanently delete" in the user interface marks an entry as "purgable", and then it will be moved to "purged" (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields "deleted", "purged", and "purgable", but nothing that says permanently deleted. -- Ravi Sanka ICS Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- On 3/13/14 1:45 PM, "Peter Cock" wrote: >Have the owners of the old datasets marked them as permanently deleted? > >Peter > >On Thu, Mar 13, 2014 at 5:35 PM, Sanka, Ravi wrote: >> Greetings, >> >> Despite being an admin, I am unable to remove old datasets from our >>Galaxy >> instance. I am following the procedure detailed in: >> >> >>https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories >>%20and%20Datasets >> >> delete_userless_histories.sh >> purge_histories.sh >> purge_libraries.sh >> purge_folders.sh >> delete_datasets.sh --> desired to remove datasets before their outer >> container had been deleted >> purge_datasets.sh >> >> None of the scripts have been changed. They all call cleanup_datasets.py >> with -d at 10 and -r enabled. >> >> But it does not appear to have any effect. All datasets (both those >>older >> than 10 days and those more recent) in /database/files are >> still present, despite the -r setting in each script. >> >> Is there some parameter or such that needs to be set in the universe >>config >> that will allow this process to work? >> >> -- >> Ravi Sanka >> ICS - Sr. Bioinformatics Engineer >> J. Craig Venter Institute >> 301-795-7743 >> -- >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
