[Dspace-tech] A Handle Body Per Community?
Hi folks I have several communities running on one dspace instance, and OAI harvesting set up. One harvester is interested in one community, the other is interested in the other- they do not want each other's data (which probably goes slightly against the nature of harvesting, but there you go)! However, because there are common dublin core elements in each schema, they will pick up bits of each other's data. I'm about to set up the handle server for this instance, but realise that I would need multiple handle bodies to filter out (in the crosswalks) the unwanted communities from each harvester. Can this be done? We are under the impression that the handle is instance-wide. If I can't have multiple handle bodies, is there another way to do this that people have tried before? Thanks very much Scott Scott Renton MIMS Project Officer Digital Library Development University Of Edinburgh 2 Buccleuch Place Edinburgh EH8 9LW 0131 651 5219 scott.ren...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] a simple question
HiI am working in a university library setting up a digital collection in Cambodia and DSpace was installed on a server before I arrived. What I have noticed with our installation is that the keywords that I add are not available in a list to chose from for a new entry. They ARE listed in the subjects listing but we often want to enter items that would have the same or very similar keywords as another entry. Does it have to do with the initial installation? With thanksShirley E Giggey in CambodiaCANADAtel: + (604) 922-8135 -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Collection item count in 1.6.2
I upgraded my DSpace 1.6.0 to 1.6.2 and replaced the default themes with my own. Everything else seems to be working fine, but the collection / community strengths are not shown. I had this configured in my themes which work fine in DSpace 1.6.0. Any hints where the problem might be ? thanks, Mika -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Collection item count in 1.6.2
On Wed, Jun 16, 2010 at 13:54, mikan.d.dspace listmail mikan.dsp...@gmail.com wrote: I upgraded my DSpace 1.6.0 to 1.6.2 and replaced the default themes with my own. Everything else seems to be working fine, but the collection / community strengths are not shown. I had this configured in my themes which work fine in DSpace 1.6.0. Any hints where the problem might be ? Just to rule out the most obvious, did you check that you updated your dspace.cfg properly? Specifically, for strengths to show this must be set: webui.strengths.show = true Also, if you have webui.strengths.cache = true you might have to run [dspace]/bin/itemcounter Regards, ~~helix84 -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Collection item count in 1.6.2
Hello Mika. If you are using xmlui, make sure that webui.strengths.cache = true in your dspace.cfg file after the upgrade. - Original Message - From: mikan.d.dspace listmail mikan.dsp...@gmail.com To: Dspace Tech dspace-tech@lists.sourceforge.net Sent: Wednesday, June 16, 2010 7:54:09 AM GMT -05:00 US/Canada Eastern Subject: [Dspace-tech] Collection item count in 1.6.2 I upgraded my DSpace 1.6.0 to 1.6.2 and replaced the default themes with my own. Everything else seems to be working fine, but the collection / community strengths are not shown. I had this configured in my themes which work fine in DSpace 1.6.0. Any hints where the problem might be ? thanks, Mika -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] DSpace 1.6.2 is released, resolves issues in 1.6.1
All, Today we’d like to announce the official release of DSpace 1.6.2! DSpace 1.6.2 can be downloaded immediately at either of the following locations: * SourceForge: https://sourceforge.net/projects/dspace/files/ * SVN: http://scm.dspace.org/svn/repo/dspace/tags/dspace-1.6.2/ DSpace 1.6.2 is a bug-fix release, which resolve several key issues in our 1.6.1 release. As mentioned previously (http://expertvoices.nsdl.org/duraspace/2010/06/11/dspace-162-release-will-fix-key-issues-in-161/), shortly after the 1.6.1 release we discovered a few key issues which required more immediate attention. We recommend anyone running DSpace 1.6.0 or 1.6.1 upgrade to 1.6.2 in order to resolve these issues. DSpace 1.6.2 does not introduce any new features, it just resolves several issues in 1.6.1 and 1.6.0. *Fixed issues* Two key issues were resolved in DSpace 1.6.2. We've fixed the broken 'start-handle-server' script (accidentally broken in the 1.6.1 release), and also resolved an issue where invalid handles (identifiers) were not being properly escaped in the 1.6.x JSPUI. The DSpace Developers felt that these two issues were important enough to warrant an immediate release. The full details of all resolved issues are available here: http://jira.dspace.org/jira/browse/DS/fixforversion/10040 *Acknowledgments* The DSpace application would not exist without the hard work and support of the community. Thanks to our early adopters of 1.6.1, who helped us to discover those few key issues we overlooked in that release. Special thanks to the small team of developers (Kim Shepherd, Keith Gilbertson, Stuart Lewis, Tim Donohue) who pulled together fixes quickly and helped with the rapid turnaround between 1.6.1 and 1.6.2. As always, thanks to Jeffrey Trimble in helping us quickly fix some key areas of the Documentation as well. A full list of people who contributed code to DSpace 1.6.2 can be found at: http://scm.dspace.org/svn/repo/dspace/tags/dspace-1.6.2/dspace/CHANGES Thanks again! Enjoy DSpace 1.6.2, and let us know what you think! Sincerely, The DSpace Developers -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Dspace smtp issues??
Greg,
I found something about this here:
http://java.sun.com/products/javamail/FAQ.html#helo
Q: I get MessagingException: 501 HELO requires domain address when trying to
send a message.
A: The SMTP provider uses the results of
InetAddress.getLocalHost().getHostName() in the SMTP HELO command. If that call
fails to return any data, no name is sent in the HELO command. Check your JDK
and name server configuration to ensure that that call returns the correct
data. You may also set the mail.smtp.localhost property to the name you want
to use for the HELO command.
It looks like the Java mail libraries are having a problem getting the name of
your host. I attached a file that makes the
InetAdress.getLocalHost().getHostname() command mentioned above so that you can
check what hostname is retrieved by this call. You can use it with these
commands: javac HostnameTest.java
java HostnameTest
and then take note of what prints out. Your system administrator may need to
adjust the settings of the name resolver on your server.
Note that DSpace 1.6+ has a configuration parameter, mail.extraproperties, that
you could use to work around this:
mail.extraproperties = mail.smtp.localhost=[yourhostname]
replacing [yourhostname] with the name of your host (no brackets).
Then you would restart Tomcat and see if everything works.
--keith
-
Hello,
RHEL 5
Dspace 1.5
postgresql
I get the following error message when trying to register a user. This message
is also very similar to what I get when clicking the Feedback link. Can anyone
point me in the right direction? I’ve looked into changing the smtp
configuration in dspace.cfg but nothing I do seems to make a difference. Do I
need to restart the server after making changes to dspace.cfg? or just restart
Tomcat?
Thanks for any help.
2010-06-15 17:14:50,280 INFO org.dspace.app.webui.servlet.RegisterServlet @
anonymous:session_id=5ABDD79E63C8CF55C8A2225AD7D09F74:ip_addr=216.183.185.5$
javax.mail.MessagingException: 501 5.0.0 HELO requires domain address
at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:1363)
at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:838)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:375)
at javax.mail.Service.connect(Service.java:275)
at javax.mail.Service.connect(Service.java:156)
at javax.mail.Service.connect(Service.java:105)
at javax.mail.Transport.send0(Transport.java:168)
at javax.mail.Transport.send(Transport.java:98)
at org.dspace.core.Email.send(Email.java:362)
at org.dspace.eperson.AccountManager.sendEmail(AccountManager.java:296)
at org.dspace.eperson.AccountManager.sendInfo(AccountManager.java:256)
at
org.dspace.eperson.AccountManager.sendRegistrationInfo(AccountManager.java:101)
at
org.dspace.app.webui.servlet.RegisterServlet.processEnterEmail(RegisterServlet.java:287)
at
org.dspace.app.webui.servlet.RegisterServlet.doDSPost(RegisterServlet.java:202)
at
org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServlet.java:147)
at org.dspace.app.webui.servlet.DSpaceServlet.doPost(DSpaceServlet.java:105)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
Greg This email message and any files transmitted with it are intended only for
the use of the individual or entity to which it is addressed and may contain
information that is privileged, confidential, and exempt from disclosure under
applicable law.� Any use, distribution, copying or disclosure by anyone other
than the intended individual or entity is prohibited without prior approval.�
If you have received this information in error, please notify the sender
immediately.import java.io.IOException;
import java.net.InetAddress;
public class HostnameTest
{
public static void main(String [] args)
{
try {
InetAddress ia
Re: [Dspace-tech] sword and authentication
Stuart, This is great info. I tried doing the servicedocument request which
works fine in the browser, but got this error. It seems like something is not
setup?
[dsp...@pocarisweat config]$ curl -i
http://blancoj.dev.deepblue.lib.umich.edu/sword/servicedocument
HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: Basic realm=SWORD
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Wed, 16 Jun 2010 20:03:48 GMT
htmlheadtitleApache Tomcat/5.5.26 - Error report/titlestyle!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--/style
/headbodyh1HTTP Status 401 - /h1HR size=1
noshade=noshadepbtype/b Status report/ppbmessage/b
u/u/ppbdescription/b uThis request requires HTTP authentication
()./u/pHR size=1 noshade=noshadeh3Apache
Tomcat/5.5.26/h3/body/html[dsp...@pocarisweat config]$
-Original Message-
From: Stuart Lewis [mailto:s.le...@auckland.ac.nz]
Sent: Friday, May 28, 2010 3:36 PM
To: Blanco, Jose
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: sword and authentication
Hi Jose,
Deposit URLS (assuming you are depositing an item into a collection) are in the
form of:
/sword/deposit/handle-id/collection
e.g.
/sword/deposit/5678/123
If you try to request that URL in a browser (HTTP verb = GET) then you will get
a HTTP 501 response with the error message you are seeing, as SWORD only
accepts POSTs to that URL. THerefore you can't interact with that page by
requesting it n a browser.
To prepare the data you'll need to create suitable packages. The following blog
post explains how this is achieved in PHP for the Facebook SWORD client:
-
http://blog.stuartlewis.com/2009/06/02/how-does-the-facebook-sword-client-actually-work/
If you want to try POSTing some files to DSpace, there are some good examples
in:
-
http://scm.dspace.org/svn/repo/dspace/tags/dspace-1.6.0/dspace-sword/README.txt
Curl is a good tool for testing, e.g.:
Authorised by dspace/dspace on a Collection:
curl -i --data-binary @dspace-sword/example/example.zip -H
Content-Disposition: filename=myDSpaceMETSItem.zip
-H Content-Type: application/zip -H X-Packaging:
http://purl.org/net/sword-types/METSDSpaceSIP;
-H X-No-Op: false -H X-Verbose: true
http://dspace:dsp...@localhost:8080/sword/deposit/123456789/2
I hope that helps,
Stuart Lewis
IT Innovations Analyst and Developer
Te Tumu Herenga The University of Auckland Library
Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
Ph: +64 (0)9 373 7599 x81928
On 29/05/2010, at 6:53 AM, Blanco, Jose wrote:
Stuart:
I was able to get pass the auth. I'm working in my dev area and I'm trying
to now use deposit to see how it would work but when I try a /deposit/handle
url I get this message a 501 error with this message:
The server does not support the functionality needed to fulfill this request
().
If I do the /servicedocument I get an xml page telling me what collections I
can deposit in, but when I try to deposit I get this error message. What url
would I need to deposit into a collection and how do I prepare the data.
-Original Message-
From: Stuart Lewis [mailto:s.le...@auckland.ac.nz]
Sent: Wednesday, May 19, 2010 5:24 PM
To: Blanco, Jose
Cc: dspace-tech@lists.sourceforge.net
Subject: RE: sword and authentication
Hi Jose,
The current SWORD specification only supports HTTP basic authentication.
DSpace will take the credentials passed this way and make use of the normal
authentication method to authentcate the user.
Authentcation is required as the service document that is returned is
customised for the user based on their authorizations in the system (e.g.
which collections they can deposit into).
The problem you will have is that CoSign is designed to be used by end users
(people), whereas SWORD authentcation is used by SWORD clients. The client
may be used by people too, but there is no way to proxy the CoSign
authentication through the SWORD client.
(This is where oAuth might be suported in future versions of the SWORD
protocol to support this use case).
One option you could have is to use a single eperson account for all
deposits, and perform authentication at your client. Your client would verify
the identity of user (using CoSign?), and then make the deposit using a
single set of credentials. To enable this you will need to make a second copy
of dspace.cfg, and change the
Re: [Dspace-tech] How to remove Register under My Account in xmlui
I figured this out. What is going wrong: I make changes in my install directory /home/dspace/dspace-1.5.2-src-release usually to /home/dspace/dspace-1.5.2-src-release/dspace/config/dspace.cfg, but sometimes to other files in /home/dspace/dspace-1.5.2-src-release/dspace. I then go to /home/dspace/dspace-1.5.2-src-release/dspace/target/dspace-1.5.2-build.dir and run ant -Dconfig=/home/dspace/dspace-1.5.2-src-release/dspace/config/dspace.cfg update and the changes I made don't get updated to /home/dspace/dspace-1.5.2-src-release/dspace/target/dspace-1.5.2-build.dir, nor to the tomcat directories mounted under /dspace. It looks like every time I make a change, I have to copy the changed files to at least 2 places for them to actually take affect. Marvin Weaver 3. How to remove Register under My Account in xmlui (Marvin Weaver) How can I remove the Register link under My Account? We do not want users to self register. I tried changing all the autoregister entries in dspace.cfg to false. dspace 1.5.2, ubuntu 8.0.4, tomcat 5.5, xmlui interface. Thank you. Marvin Weaver -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] sword and authentication
I see now. I need to use the -u for the user name. I'm new to curl, but
getting the idea now.
-Original Message-
From: Blanco, Jose [mailto:blan...@umich.edu]
Sent: Wednesday, June 16, 2010 4:05 PM
To: Stuart Lewis
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] sword and authentication
Stuart, This is great info. I tried doing the servicedocument request which
works fine in the browser, but got this error. It seems like something is not
setup?
[dsp...@pocarisweat config]$ curl -i
http://blancoj.dev.deepblue.lib.umich.edu/sword/servicedocument
HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: Basic realm=SWORD
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Wed, 16 Jun 2010 20:03:48 GMT
htmlheadtitleApache Tomcat/5.5.26 - Error report/titlestyle!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--/style
/headbodyh1HTTP Status 401 - /h1HR size=1
noshade=noshadepbtype/b Status report/ppbmessage/b
u/u/ppbdescription/b uThis request requires HTTP authentication
()./u/pHR size=1 noshade=noshadeh3Apache
Tomcat/5.5.26/h3/body/html[dsp...@pocarisweat config]$
-Original Message-
From: Stuart Lewis [mailto:s.le...@auckland.ac.nz]
Sent: Friday, May 28, 2010 3:36 PM
To: Blanco, Jose
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: sword and authentication
Hi Jose,
Deposit URLS (assuming you are depositing an item into a collection) are in the
form of:
/sword/deposit/handle-id/collection
e.g.
/sword/deposit/5678/123
If you try to request that URL in a browser (HTTP verb = GET) then you will get
a HTTP 501 response with the error message you are seeing, as SWORD only
accepts POSTs to that URL. THerefore you can't interact with that page by
requesting it n a browser.
To prepare the data you'll need to create suitable packages. The following blog
post explains how this is achieved in PHP for the Facebook SWORD client:
-
http://blog.stuartlewis.com/2009/06/02/how-does-the-facebook-sword-client-actually-work/
If you want to try POSTing some files to DSpace, there are some good examples
in:
-
http://scm.dspace.org/svn/repo/dspace/tags/dspace-1.6.0/dspace-sword/README.txt
Curl is a good tool for testing, e.g.:
Authorised by dspace/dspace on a Collection:
curl -i --data-binary @dspace-sword/example/example.zip -H
Content-Disposition: filename=myDSpaceMETSItem.zip
-H Content-Type: application/zip -H X-Packaging:
http://purl.org/net/sword-types/METSDSpaceSIP;
-H X-No-Op: false -H X-Verbose: true
http://dspace:dsp...@localhost:8080/sword/deposit/123456789/2
I hope that helps,
Stuart Lewis
IT Innovations Analyst and Developer
Te Tumu Herenga The University of Auckland Library
Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
Ph: +64 (0)9 373 7599 x81928
On 29/05/2010, at 6:53 AM, Blanco, Jose wrote:
Stuart:
I was able to get pass the auth. I'm working in my dev area and I'm trying
to now use deposit to see how it would work but when I try a /deposit/handle
url I get this message a 501 error with this message:
The server does not support the functionality needed to fulfill this request
().
If I do the /servicedocument I get an xml page telling me what collections I
can deposit in, but when I try to deposit I get this error message. What url
would I need to deposit into a collection and how do I prepare the data.
-Original Message-
From: Stuart Lewis [mailto:s.le...@auckland.ac.nz]
Sent: Wednesday, May 19, 2010 5:24 PM
To: Blanco, Jose
Cc: dspace-tech@lists.sourceforge.net
Subject: RE: sword and authentication
Hi Jose,
The current SWORD specification only supports HTTP basic authentication.
DSpace will take the credentials passed this way and make use of the normal
authentication method to authentcate the user.
Authentcation is required as the service document that is returned is
customised for the user based on their authorizations in the system (e.g.
which collections they can deposit into).
The problem you will have is that CoSign is designed to be used by end users
(people), whereas SWORD authentcation is used by SWORD clients. The client
may be used by people too, but there is no way to proxy the CoSign
authentication through the SWORD client.
(This is where oAuth might be suported in future versions of the SWORD
protocol to support this use case).
One option you could
[Dspace-tech] DSpace Installation/Setup Assistance
Greetings! I'm interested in learning how to install and implement DSpace at my library for a digital photo repository we would like to create. I have thumbed through main of the installation guides provided on the DSpace.org website however I'm not 100% sure of what is needed to get DSpace up and running. I have some experience with Ubuntu server but would like to know if there is an easy how to installation guide. Any help would be greatly appreciated!! Thank You! Adam Day Network Administrator Twin Falls Public Library http://www.twinfallspubliclibrary.org -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] solr crashing dspace 1.6
Hi Charles, I wanted to follow up to see if you've been able to make any progress with your memory problems with SOLR. Our instance has quite a bit of traffic, so we would get errors like this: 2010-06-16 10:37:15,840 ERROR org.dspace.statistics.SolrLogger @ Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. request: http://localhost:8080/solr/statistics/update org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. request: http://localhost:8080/solr/statistics/update at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183) at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217) at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:85) at org.dspace.statistics.SolrLogger.post(SolrLogger.java:247) At this point I was thinking my only option was to throw more resources at the problem. Thus my previous comment to give it more memory, as it does look like the servers gets overloaded with SOLR requests. However, even still it then slowly starts to fall behind on servicing every request, thus eventually running out of searchers. Something one would call a cascading problem. Anyways, I looked into it, and SOLR has a much more elegant, and cheaper way of fixing this problem. And I even saw in the code where the author of the DSpace solr code Mark Diggory / Ben Bosman et al. left a commenthttp://fisheye3.atlassian.com/browse/dspace/dspace/tags/dspace-1.6.1/dspace-stats/src/main/java/org/dspace/statistics/SolrLogger.java?r=HEAD#l246 that knew this was coming. solr.add(doc1); // TODO: requires further load testing, very fast commits might cause issues solr.commit(false, false); Thus immediately after every UsageEvent, something that GoogleBot / visitors do about 5 times / second. It will add the request to the SOLR's queue ( a cheap operation), and then immediately try to commit (imaginably an expensive operation). This is bad, as the comment says. To remedy this, I've commented out the solr.commit in SolrLogger.java, and instead rely upon SOLR to figure out when to do commits. autoCommit. Essentially just uncomment the code part for autoCommit in solrconfig.xml http://fisheye3.atlassian.com/browse/dspace/dspace/tags/dspace-1.6.1/dspace/solr/statistics/conf/solrconfig.xml?r=HEAD#l143 maxDocs - number of updates since last commit is greater than this maxTime - oldest uncommited update (in ms) is this long ago autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit After these changes I went from about two SOLR errors per second, to zero / since. A 5183904902% improvement. I did some further stress testing, and I couldn't cause a SOLR error after this, even while cranking it up to about 60+ requests per second. Another thing to note was that during the constant error storm, our CPU usage was hovering at about 95%, now its dropped to a sane level at about 10%. If this works for everyone, I'll commit it to trunk. Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Mon, May 10, 2010 at 5:30 AM, TAYLOR Robin robin.tay...@ed.ac.uk wrote: Hi Bill, I have found JMX/jconsolve invaluable when investigating memory problems http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html. Cheers, Robin. Robin Taylor Main Library University of Edinburgh Tel. 0131 6513808 -Original Message- From: Anderson, Charles W [mailto:bill.ander...@library.gatech.edu] Sent: 06 May 2010 16:41 To: dspace-tech Subject: [Dspace-tech] solr crashing dspace 1.6 We're on our second attempt to upgrade from 1.5.2 to 1.6. We have been plagued with m memory errors both times, and we' think we've narrowed it down to something that's going on in the solr webapp. We've got solr running in it's very own Tomcat instance; which has helped, but hasn't completely eliminated the problems. Just before we crash, we're seeing lots and lots of the following in the dspace log: org.apache.solr.common.SolrException: Internal Server Error Then, eventually, we start getting this: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error: Timeout waiting for idle object Meanwhile, over in catalina.out, we have these, which seem to correspond to the solr errors in the dspace log: SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. And, eventually, the voice of doom: Exception in thread TP-Processor550
