Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
Thanks, Ravi Peter I've added a card to get the allow_user_dataset_purge options into the client and to better show the viable options to the user: https://trello.com/c/RCPZ9zMF On Fri, Mar 14, 2014 at 11:10 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi rsa...@jcvi.org wrote: I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using permanently delete in the user interface marks an entry as purgable, and then it will be moved to purged (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Hmm. Right now I've unable (via the web interface) to permanently delete a history - it stays stuck as deleted, and thus (presumably) won't get purged by the clean up scripts. I've tried: 1. Load problem history 2. Rename the history DIE DIE to avoid confusion 3. Top right menu, Delete permanently 4. Prompted Really delete the current history permanently? This cannot be undone, OK 5. Told History deleted, a new history is active 6. Top right menu, Saved Histories 7. Click Advanced Search, status all 8. Observe DIE DIE history is only deleted (while other older histories are deleted permanently) (BAD) 9. Run the cleanup scripts, $ sh scripts/cleanup_datasets/delete_userless_histories.sh $ sh scripts/cleanup_datasets/purge_histories.sh $ sh scripts/cleanup_datasets/purge_libraries.sh $ sh scripts/cleanup_datasets/purge_folders.sh $ sh scripts/cleanup_datasets/purge_datasets.sh 10. Reload the saved history list, no change. 11. Using the drop down menu, select Delete Permanently 12. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 13. No change to history status (BAD) 14. Tick the check-box, and use the Delete Permanently button at the bottom of the page 15. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 16. No change to history status (BAD) 17. Run the cleanup scripts, no change. Note that in my universe_wsgi.ini I have not (yet) set: allow_user_dataset_purge = True If this setting is important, then the interface seems confused - and if quotas are enforced, very frustrating :( Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
I believe it's a (BAD) silent failure mode in the server code. If I understand correctly, the purge request isn't coughing an error when it gets to the 'allow_user_dataset_purge' check and instead is silently marking (or re-marking) the datasets as deleted. I would rather it fail with a 403 error if purge is explicitly requested. That said, it of course would be better to remove the purge operation based on the configuration then to show an error after we've found you can't do the operation. The same holds true for the 'permanently remove this dataset' link in deleted datasets. I'll see if I can find out the answer to your question on the cleanup scripts. On Tue, Mar 18, 2014 at 10:49 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Tue, Mar 18, 2014 at 2:14 PM, Carl Eberhard carlfeberh...@gmail.com wrote: Thanks, Ravi Peter I've added a card to get the allow_user_dataset_purge options into the client and to better show the viable options to the user: https://trello.com/c/RCPZ9zMF Thanks Carl - so this was a user interface bug, showing the user non-functional permanent delete (purge) options. That's clearer now. In this situation can the user just 'delete', and wait N days for the cleanup scripts to actually purge the files and free the space? (It seems N=10 in scripts/cleanup/purge_*.sh at least, elsewhere like the underlying Python script the default looks like N=60). Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
The cleanup scripts enforce a sort of lifetime for the datasets. The first time they're run, they may mark a dataset as deleted and also reset the update time and you'll have to wait N days for the next stage of the lifetime. The next time they're run, or if a dataset has already been marked as deleted, the actual file removal happens and purged is set to true (if it wasn't already). You can manually pass in '-d 0' to force removal of datasets recently marked as deleted. The purge scripts do not check 'allow_user_dataset_purge', of course. On Tue, Mar 18, 2014 at 11:50 AM, Carl Eberhard carlfeberh...@gmail.comwrote: I believe it's a (BAD) silent failure mode in the server code. If I understand correctly, the purge request isn't coughing an error when it gets to the 'allow_user_dataset_purge' check and instead is silently marking (or re-marking) the datasets as deleted. I would rather it fail with a 403 error if purge is explicitly requested. That said, it of course would be better to remove the purge operation based on the configuration then to show an error after we've found you can't do the operation. The same holds true for the 'permanently remove this dataset' link in deleted datasets. I'll see if I can find out the answer to your question on the cleanup scripts. On Tue, Mar 18, 2014 at 10:49 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Tue, Mar 18, 2014 at 2:14 PM, Carl Eberhard carlfeberh...@gmail.com wrote: Thanks, Ravi Peter I've added a card to get the allow_user_dataset_purge options into the client and to better show the viable options to the user: https://trello.com/c/RCPZ9zMF Thanks Carl - so this was a user interface bug, showing the user non-functional permanent delete (purge) options. That's clearer now. In this situation can the user just 'delete', and wait N days for the cleanup scripts to actually purge the files and free the space? (It seems N=10 in scripts/cleanup/purge_*.sh at least, elsewhere like the underlying Python script the default looks like N=60). Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi rsa...@jcvi.org wrote: I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using permanently delete in the user interface marks an entry as purgable, and then it will be moved to purged (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
On Fri, Mar 14, 2014 at 11:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 13, 2014 at 6:40 PM, Sanka, Ravi rsa...@jcvi.org wrote: I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. I would welcome clarification from the Galaxy Team, here and on the wiki page which might benefit from a flow diagram? https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets My assumption is using permanently delete in the user interface marks an entry as purgable, and then it will be moved to purged (and the associated file on disk deleted) by the cleanup scripts - but I'm a bit hazy on this any why it takes a while for a user's usage figures to change. Hmm. Right now I've unable (via the web interface) to permanently delete a history - it stays stuck as deleted, and thus (presumably) won't get purged by the clean up scripts. I've tried: 1. Load problem history 2. Rename the history DIE DIE to avoid confusion 3. Top right menu, Delete permanently 4. Prompted Really delete the current history permanently? This cannot be undone, OK 5. Told History deleted, a new history is active 6. Top right menu, Saved Histories 7. Click Advanced Search, status all 8. Observe DIE DIE history is only deleted (while other older histories are deleted permanently) (BAD) 9. Run the cleanup scripts, $ sh scripts/cleanup_datasets/delete_userless_histories.sh $ sh scripts/cleanup_datasets/purge_histories.sh $ sh scripts/cleanup_datasets/purge_libraries.sh $ sh scripts/cleanup_datasets/purge_folders.sh $ sh scripts/cleanup_datasets/purge_datasets.sh 10. Reload the saved history list, no change. 11. Using the drop down menu, select Delete Permanently 12. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 13. No change to history status (BAD) 14. Tick the check-box, and use the Delete Permanently button at the bottom of the page 15. Prompted History contents will be removed from disk, this cannot be undone. Continue, OK 16. No change to history status (BAD) 17. Run the cleanup scripts, no change. Note that in my universe_wsgi.ini I have not (yet) set: allow_user_dataset_purge = True If this setting is important, then the interface seems confused - and if quotas are enforced, very frustrating :( Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Unable to remove old datasets
I do not think so. Several individual datasets have been deleted (clicked the upper-right X on the history item box) but no History has been permanently deleted. Is there any indication in the database if target dataset or datasets were marked for permanent deletion? In the dataset table, I see fields deleted, purged, and purgable, but nothing that says permanently deleted. -- Ravi Sanka ICS Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- On 3/13/14 1:45 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Have the owners of the old datasets marked them as permanently deleted? Peter On Thu, Mar 13, 2014 at 5:35 PM, Sanka, Ravi rsa...@jcvi.org wrote: Greetings, Despite being an admin, I am unable to remove old datasets from our Galaxy instance. I am following the procedure detailed in: https://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories %20and%20Datasets delete_userless_histories.sh purge_histories.sh purge_libraries.sh purge_folders.sh delete_datasets.sh -- desired to remove datasets before their outer container had been deleted purge_datasets.sh None of the scripts have been changed. They all call cleanup_datasets.py with -d at 10 and -r enabled. But it does not appear to have any effect. All datasets (both those older than 10 days and those more recent) in galaxy root/database/files are still present, despite the -r setting in each script. Is there some parameter or such that needs to be set in the universe config that will allow this process to work? -- Ravi Sanka ICS - Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
