Re: [galaxy-dev] Bug importing from data library to history
I believe I may have found the bug... Line 258 of cleanup_datasets.py is the query for library_dataset_ids when "force_retry" is set. While it correctly eliminates the the app.model.LibraryDataset.table.c.purged == False from the where clause, it incorrectly eliminates the app.model.LibraryDataset.table.c.deleted == True. My suggested fix would be to correct that query by adding app.model.LibraryDataset.table.c.deleted == True to the where clause. Cleaning up users databases might be a bit trickier, but I think you could find the affected library datasets by the following query: select * from library_dataset where deleted=f and purged=t; With those, I would think setting purged to f and then setting deleted = f for the associated library_dataset_dataset_association records. Lance Nate Coraor wrote: Lance Parsons wrote: Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Any community offers to debug/fix are always happily welcomed. =) My reply to Alex: See line 1925 in lib/galaxy/web/controllers/library_common.py. Try adding a bit of debugging to find out if hda.deleted == True when the copied HistoryDatasetAssociation is created, and whether the source LibraryDatasetDatasetAssociation is itself deleted (which should be the only way the resulting hda could be deleted). I'll add that a key to figuring this out is probably is determining what's different about these older library items and the newer ones that don't exhibit the problem (if this is also true in your environment as it is in his). If no answers turn up in a few days, then we can open an issue for it in Bitbucket. --nate Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Lance Parsons wrote: > Good catch, that does indeed seem to be the issue. And it only > seems to happen with my older data libraries. > > I didn't see a bug report for this, but I'd be happy to file one > and/or take a stab at a fix if someone can point me in the right > direction in the code. Any community offers to debug/fix are always happily welcomed. =) My reply to Alex: See line 1925 in lib/galaxy/web/controllers/library_common.py. Try adding a bit of debugging to find out if hda.deleted == True when the copied HistoryDatasetAssociation is created, and whether the source LibraryDatasetDatasetAssociation is itself deleted (which should be the only way the resulting hda could be deleted). I'll add that a key to figuring this out is probably is determining what's different about these older library items and the newer ones that don't exhibit the problem (if this is also true in your environment as it is in his). If no answers turn up in a few days, then we can open an issue for it in Bitbucket. --nate > > Lance > > Nate Coraor wrote: > >Lance Parsons wrote: > >>I'm running a local instance of Galaxy that has been working quite > >>well. However, I've recently run into a problem when importing > >>datasets from a data library into a history. The Size on Disk of > >>the history grows by the size of the imported dataset, however, the > >>new dataset is not visible (and the number of datasets does not > >>change in the saved histories view). > >> > >>Any thoughts on what might be going on here? Thanks. > > > >Hey Lance, > > > >No real ideas, but Alex Bossers just posted up saying that old library > >datasets are importing into his histories as deleted. Could you check > >and see if this is the case in your instance as well? > > > >Thanks, > >--nate > > > >>-- > >>Lance Parsons - Scientific Programmer > >>134 Carl C. Icahn Laboratory > >>Lewis-Sigler Institute for Integrative Genomics > >>Princeton University > >> > >>___ > >>Please keep all replies on the list by using "reply all" > >>in your mail client. To manage your subscriptions to this > >>and other Galaxy lists, please use the interface at: > >> > >> http://lists.bx.psu.edu/ > > -- > Lance Parsons - Scientific Programmer > 134 Carl C. Icahn Laboratory > Lewis-Sigler Institute for Integrative Genomics > Princeton University > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Ah, sorry, I just saw your note to Alex regarding the pieces of code to check out. I'll start doing some digging and let you know if I find anything out. See line 1925 in lib/galaxy/web/controllers/library_common.py. Try adding a bit of debugging to find out if hda.deleted == True when the copied HistoryDatasetAssociation is created, and whether the source LibraryDatasetDatasetAssociation is itself deleted (which should be the only way the resulting hda could be deleted). Lance Lance Parsons wrote: Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Lance Parsons wrote: > I'm running a local instance of Galaxy that has been working quite > well. However, I've recently run into a problem when importing > datasets from a data library into a history. The Size on Disk of > the history grows by the size of the imported dataset, however, the > new dataset is not visible (and the number of datasets does not > change in the saved histories view). > > Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate > > -- > Lance Parsons - Scientific Programmer > 134 Carl C. Icahn Laboratory > Lewis-Sigler Institute for Integrative Genomics > Princeton University > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Bug importing from data library to history
I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
