Re: [Dspace-tech] Asset store cleanup failing
Yes, that was exactly how we corrected ours! I never figured out where the bad data came from, but once I cleaned it up we've never had a problem with it since. I don't think you've broken anything either as I don't think the primary_bitstream column is used in DSpace anymore? Oh, and we do have cleanup in our cron table to run nightly. Sue From: younga3...@gmail.com [mailto:younga3...@gmail.com] On Behalf Of Anjanette Young Sent: Friday, December 04, 2009 1:30 PM To: Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Susan, Thank you for responding. Yes it was the same error. The logs in production were too noisy and not set to debug, so I did not have a good log of the error. Once I figured out the how to remove the red-herring error (different # of assetstores on dev) on a partial copy of production on our development server, the referential integrity violation error became apparent. Error: ERROR: primary_bitstream_id_fk referential integrity violation - key in bitstream still referenced from bundle The SQL I used to find the problem bitstreams: SELECT bitstream.bitstream_id, bitstream.deleted, bundle.bundle_id FROM bitstream JOIN bundle ON bundle.primary_bitstream_id = bitstream.bitstream_id WHERE bundle.primary_bitstream_id 0 AND bitstream.deleted = true; Then updated those primary_bitstream_id's to NULL. Hope I have not broken anything. That this problem is occurring in 1.5.2 is probably evidence of our lack of repository hygiene: we ought to add bin/cleanup to cron. We don't delete items out of the repository very often, so it was probably not added before. Thank you, Anjanette On Fri, Dec 4, 2009 at 9:44 AM, Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov wrote: Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.demailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Asset store cleanup failing
Hello all, most probably the bad data came from deleting an item. I noticed that users got confused. They take the flag for primary bitstream as a flag for deletion. So they check the bitstream they want to delete as primary bistream and then delete it. Hope that helps Claudia Jürgen Yes, that was exactly how we corrected ours! I never figured out where the bad data came from, but once I cleaned it up we've never had a problem with it since. I don't think you've broken anything either as I don't think the primary_bitstream column is used in DSpace anymore? Oh, and we do have cleanup in our cron table to run nightly. Sue From: younga3...@gmail.com [mailto:younga3...@gmail.com] On Behalf Of Anjanette Young Sent: Friday, December 04, 2009 1:30 PM To: Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Susan, Thank you for responding. Yes it was the same error. The logs in production were too noisy and not set to debug, so I did not have a good log of the error. Once I figured out the how to remove the red-herring error (different # of assetstores on dev) on a partial copy of production on our development server, the referential integrity violation error became apparent. Error: ERROR: primary_bitstream_id_fk referential integrity violation - key in bitstream still referenced from bundle The SQL I used to find the problem bitstreams: SELECT bitstream.bitstream_id, bitstream.deleted, bundle.bundle_id FROM bitstream JOIN bundle ON bundle.primary_bitstream_id = bitstream.bitstream_id WHERE bundle.primary_bitstream_id 0 AND bitstream.deleted = true; Then updated those primary_bitstream_id's to NULL. Hope I have not broken anything. That this problem is occurring in 1.5.2 is probably evidence of our lack of repository hygiene: we ought to add bin/cleanup to cron. We don't delete items out of the repository very often, so it was probably not added before. Thank you, Anjanette On Fri, Dec 4, 2009 at 9:44 AM, Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov wrote: Seems like we had this problem back in 1.4.2. Ours did the same thing - the script output shows the success messages you mention below, however if you look in the dspace.log, you'll see that it's indeed getting relational integrity errors - can't remember exactly, but it was something between the bitstream, bundle2bitstream, and bundle tables - it might have been the primary_bitstream_id was pointing to the wrong row in the bundle2bitstream and/or bitstream tables..?? In order to correct it, I believe I had to write some sql queries and, one by one, clean up the errors by updating column(s) in the offending table. Sorry I can't remember exactly what I did, but if you'll look in the dspace.log, you'll see the output from the cleanup script and it should give you a better idea of exactly what needs to be done. If you'll check your log and post the error(s) you're getting, it might help me remember what exactly we did to fix it. Best, Sue Walker-Thornton -Original Message- From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.demailto:claudia.juer...@ub.tu-dortmund.de] Sent: Wednesday, December 02, 2009 7:26 AM To: Anjanette Young Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Asset store cleanup failing Hello Anjanette, can you provide more information from the logs, postgres and/or dspace? Claudia Jürgen Anjanette Young schrieb: We are running DSpace 1.5.2 with Postgresql 7.3. I have been running $dspace/bin/cleanup which happily states: Cleaning the asset store Cleanup completed But files have not been cleaned from the assetstore, nor have the deleted=true bitstreams been deleted from the db table. I've checked to see if it might be related to http://jira.dspace.org/jira/browse/DS-197 , but none of the primary_bitstream_id 0 bundles are related to bitstreams marked for deletion. Has anyone else run into this problem? Any help appreciated, Anjanette -- Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Embargo in 1.6
Hi Stuart:
I'll take a crack at some of your questions: see remarks inline below.
Thanks,
Richard R
On Tue, 2009-12-08 at 17:40 +1300, stuart yeates wrote:
I have some questions about the Embargo plugin in 1.6. I'm basing this
on http://wiki.dspace.org/index.php/Embargo_1.6 and trolling through the
subversion repository (dspace-1.6.0-rc1 tag).
We'd like to have an drop-down box in our self-deposit which allows
users to select an embargo period (probably 3, 6, 12, 18 or 24 months).
This then gets put in the metadata field pointed to by
embargo.field.terms (probably 'VUW.embargo'), and the date of uplift
calculated and stored in that pointed to by embargo.field.lift (probably
'available').
As far as submission goes, you could do this fairly easily by using the
configurations available in 'input-forms.xml' by defining the drop-down
list and it's values. See the doc for more information (all XML, no
Java)
The DefaultEmbargoSetter automatically sets the default permissions so
that while the item metadata for an embargoed item is globally readable,
the bitstreams are inaccessible to everyone but admins. [This could be
overridden to a less strict lockdown by overriding the setEmbargo
method, we havent' thrashed this out yet] DefaultEmbargoSetter also
calculates the embargo.field.lift date, from the embargo.field.terms and
the current time/date.
Once a day the EmbargoManager runs and uplifts items whose embargo has
expired. Uplifting involves setting the permissions to whatever the
default permissions are for the collection it's in, making item's
bitstreams public.
My questions are:
[1] Does the above sound sane?
Yes, that's the general flow of embargo processing, and what you want to
do falls generally within it.
[2] Is there any way to generate notifications of lifting? The easiest
thing I can see would be to do a search for the embargo.field.terms
field, sorting on the availability, and supply that as an RSS feed.
You could do something like that, but you could also directly act when
the embargo is lifted. On the wiki page you cite, look at the Downloads
section of the Prototype implementation for 'Embargo-1.6-new.zip'. This
contains some example code from Harvard that extends the EmbargoLifter
to send an email to the submitter whenever an embargoed item is lifted.
This could give you an idea of how to do something similar (you didn't
characterize how the notification target(s) would be identified, so I
can't be more specific).
[3] We're considering how to direct users to other sources for the item
when it's currently embargoed. This would probably be involve displaying
a block of text which might be inviting them to login and giving them
alternative access routes to the item. Has anyone done this?
This might be a bit more involved, especially if (as seems likely by
your description) the text and access route would be item-specific.
Having said this, if you could represent it in a metadata field or
fields that would be entered at submission, your Lifter could
programmatically remove them when the embargo ends.
[4] In the wiki at http://wiki.dspace.org/index.php/Embargo_1.6, if I am
reading things correctly, the second to last option in the config file
snippet is missing the relevant default option and the last option has
it truncated. Am I reading it correctly?
The wiki page does look a little truncated: go with what you find in the
dspace.cfg that ships with the 1.6 distribution and current tech doc.
Due to my technical skills, I'd prefer options that involve XSLT to
those involving Java :)
Except where noted above in [2] and [3] above about customizing the
Lifter, the only Java coding you would need to do is in the
EmbargoSetter (to interpret your fixed intervals into DSpace dates), but
this is fairly simple (very rough code, just an example):
[in 'parseTerms' method]
...
if (3 months.equals(terms)) {
long monMillis = 90 * 24 * 60 * 60 * 1000;
return new DCDate(new Date(System.currentTimeMillis() + monMillis));
} else if (6 months.equals(terms)) {
.
cheers
stuart
--
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] idle in transaction workaround for Solaris
I created a Solaris version of Cory Snavely's workaround from:
http://wiki.dspace.org/index.php/Idle_In_Transaction_Problem
To see detailed process information:
1. you must have update_process_title = on in postgresql.conf
2. be the owner of the process or root
Solaris platform specifics from the PostgreSQL documentation:
Solaris requires special handling. You must use /usr/ucb/ps, rather than
/bin/ps. You also must use two w flags, not just one. In addition, your
original invocation of the postgres command must have a shorter ps status
display than that provided by each server process. If you fail to do all three
things, the ps output for each server process will be the original postgres
command line.
Since pgrep and pkill are built on /usr/bin/ps (not /usr/ucb/ps) Cory's
workaround didn't work for me.
I created my own version that does not rely on pgrep or pkill:
/usr/bin/test ` /usr/ucb/ps auxwww | grep '[i]dle in transaction' | \
/usr/bin/wc -l ` -gt 20 kill -9 `/usr/ucb/ps auxwww | \
grep '[i]dle in transaction' | \
awk '{if(min==){min=$7}; if($7 min) {min=$7; oldest_id=$2};} END {print
oldest_id}'`
I'm using this on Solaris 10. If anyone else with Solaris would like to try it
out, please let me know the results.
Thanks,
Damian Marinaccio
The Wallace Center
90 Lomb Memorial Drive
Rochester, NY 14623
585-475-7741
dxm...@rit.edu
CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.
--
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace 1.6 Testathon - The Big Bug Hunt - Report for Day 1
Hi Stuart and all, Thanks for your advice. I logined to the http://eldorado2.tu-dortmund.de:8080/jspui one of the Testathon and found that I have already been assigned as administrator. There are a couple of points I would like to share here: 1. I tried both JSPUI and XMLUI of the same site. It surprised me that I had to registered for each UI separately. If I am not wrong, in previous version, there is no need to register to each UI. 2. I browse the statistics by clicking the view statistics button of a particular community or collection. The Top Country Views is fine, but the Top City Views needs more improvement. I am living in Hat Yai city or Songkhla Province, but the count is associated to Bangkok which is not correct. How does this module work ? Does it retrieve city information from the IP or just use the capital city of the country ? 3. I also tried the OAI and SWORD, seems that these are still underconstruction. 4. One question, what is the MIT Links at the bottom of each item. For example, in http://eldorado2.tu-dortmund.de:8080/jspui/handle/123456789/11 there is a button label MIT li...@sfx which points to http://sfx.hbz-nrw.de/sfx_ubd?title=Test+submission+with+workflow+step+1aulast=Doeaufirst=Jemimadate=2009-12-07 Is it specific to the site only ? Panyarak Ngamsritragul On Tue, 8 Dec 2009, Stuart Lewis wrote: Hi Panyarak, I did subscribe to a test site a few days ago, could not do anything but browse. I wanted to try some statistics, submitting some items, but no authority to do that... If you need admin access to one of the test systems, send an email to this email list with the system URL and your email address, and we'll make sure you get admin access granted. Many thanks for helping with the testathon, Stuart Lewis IT Innovations Analyst and Developer Te Tumu Herenga The University of Auckland Library Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand Ph: 64 9 373-7599 x81928 http://www.library.auckland.ac.nz/ -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
