Re: [Dspace-tech] Asset store cleanup failing
Yes, that was exactly how we corrected ours! I never figured out where the bad data came from, but once I cleaned it up we've never had a problem with it since. I don't think you've broken anything either as I don't think the primary_bitstream column is used in DSpace anymore? Oh, and we do have cleanup in our cron table to run nightly. Sue From: younga3...@gmail.com [mailto:younga3...@gmail.com] On Behalf Of Anjanette Young Sent: Friday, December 04, 2009 1:30 PM To: Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Susan, Thank you for responding. Yes it was the same error. The logs in production were too noisy and not set to debug, so I did not have a good log of the error. Once I figured out the how to remove the red-herring error (different # of assetstores on dev) on a partial copy of production on our development server, the referential integrity violation error became apparent. Error: ERROR: primary_bitstream_id_fk referential integrity violation - key in bitstream still referenced from bundle The SQL I used to find the problem bitstreams: SELECT bitstream.bitstream_id, bitstream.deleted, bundle.bundle_id FROM bitstream JOIN bundle ON bundle.primary_bitstream_id = bitstream.bitstream_id WHERE bundle.primary_bitstream_id 0 AND bitstream.deleted = true; Then updated those primary_bitstream_id's to NULL. Hope I have not broken anything. That this problem is occurring in 1.5.2 is probably evidence of our lack of repository hygiene: we ought to add bin/cleanup to cron. We don't delete items out of the repository very often, so it was probably not added before. Thank you, Anjanette On Fri, Dec 4, 2009 at 9:44 AM, Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov wrote: Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.demailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Asset store cleanup failing
Hello all, most probably the bad data came from deleting an item. I noticed that users got confused. They take the flag for primary bitstream as a flag for deletion. So they check the bitstream they want to delete as primary bistream and then delete it. Hope that helps Claudia Jürgen Yes, that was exactly how we corrected ours! I never figured out where the bad data came from, but once I cleaned it up we've never had a problem with it since. I don't think you've broken anything either as I don't think the primary_bitstream column is used in DSpace anymore? Oh, and we do have cleanup in our cron table to run nightly. Sue From: younga3...@gmail.com [mailto:younga3...@gmail.com] On Behalf Of Anjanette Young Sent: Friday, December 04, 2009 1:30 PM To: Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Susan, Thank you for responding. Yes it was the same error. The logs in production were too noisy and not set to debug, so I did not have a good log of the error. Once I figured out the how to remove the red-herring error (different # of assetstores on dev) on a partial copy of production on our development server, the referential integrity violation error became apparent. Error: ERROR: primary_bitstream_id_fk referential integrity violation - key in bitstream still referenced from bundle The SQL I used to find the problem bitstreams: SELECT bitstream.bitstream_id, bitstream.deleted, bundle.bundle_id FROM bitstream JOIN bundle ON bundle.primary_bitstream_id = bitstream.bitstream_id WHERE bundle.primary_bitstream_id 0 AND bitstream.deleted = true; Then updated those primary_bitstream_id's to NULL. Hope I have not broken anything. That this problem is occurring in 1.5.2 is probably evidence of our lack of repository hygiene: we ought to add bin/cleanup to cron. We don't delete items out of the repository very often, so it was probably not added before. Thank you, Anjanette On Fri, Dec 4, 2009 at 9:44 AM, Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov wrote: Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.demailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Asset store cleanup failing
Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Asset store cleanup failing
Susan, Thank you for responding. Yes it was the same error. The logs in production were too noisy and not set to debug, so I did not have a good log of the error. Once I figured out the how to remove the red-herring error (different # of assetstores on dev) on a partial copy of production on our development server, the referential integrity violation error became apparent. Error: ERROR: primary_bitstream_id_fk referential integrity violation - key in bitstream still referenced from bundle The SQL I used to find the problem bitstreams: SELECT bitstream.bitstream_id, bitstream.deleted, bundle.bundle_id FROM bitstream JOIN bundle ON bundle.primary_bitstream_id = bitstream.bitstream_id WHERE bundle.primary_bitstream_id 0 AND bitstream.deleted = true; Then updated those primary_bitstream_id's to NULL. Hope I have not broken anything. That this problem is occurring in 1.5.2 is probably evidence of our lack of repository hygiene: we ought to add bin/cleanup to cron. We don't delete items out of the repository very often, so it was probably not added before. Thank you, Anjanette On Fri, Dec 4, 2009 at 9:44 AM, Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] susan.m.thorn...@nasa.gov wrote: Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Asset store cleanup failing
We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
