Re: [Dspace-tech] solr crashing dspace 1.6
Hi Charles, I wanted to follow up to see if you've been able to make any progress with your memory problems with SOLR. Our instance has quite a bit of traffic, so we would get errors like this: 2010-06-16 10:37:15,840 ERROR org.dspace.statistics.SolrLogger @ Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. request: http://localhost:8080/solr/statistics/update org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. request: http://localhost:8080/solr/statistics/update at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183) at org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217) at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:85) at org.dspace.statistics.SolrLogger.post(SolrLogger.java:247) At this point I was thinking my only option was to throw more resources at the problem. Thus my previous comment to give it more memory, as it does look like the servers gets overloaded with SOLR requests. However, even still it then slowly starts to fall behind on servicing every request, thus eventually running out of searchers. Something one would call a cascading problem. Anyways, I looked into it, and SOLR has a much more elegant, and cheaper way of fixing this problem. And I even saw in the code where the author of the DSpace solr code Mark Diggory / Ben Bosman et al. left a commenthttp://fisheye3.atlassian.com/browse/dspace/dspace/tags/dspace-1.6.1/dspace-stats/src/main/java/org/dspace/statistics/SolrLogger.java?r=HEAD#l246 that knew this was coming. solr.add(doc1); // TODO: requires further load testing, very fast commits might cause issues solr.commit(false, false); Thus immediately after every UsageEvent, something that GoogleBot / visitors do about 5 times / second. It will add the request to the SOLR's queue ( a cheap operation), and then immediately try to commit (imaginably an expensive operation). This is bad, as the comment says. To remedy this, I've commented out the solr.commit in SolrLogger.java, and instead rely upon SOLR to figure out when to do commits. autoCommit. Essentially just uncomment the code part for autoCommit in solrconfig.xml http://fisheye3.atlassian.com/browse/dspace/dspace/tags/dspace-1.6.1/dspace/solr/statistics/conf/solrconfig.xml?r=HEAD#l143 maxDocs - number of updates since last commit is greater than this maxTime - oldest uncommited update (in ms) is this long ago autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit After these changes I went from about two SOLR errors per second, to zero / since. A 5183904902% improvement. I did some further stress testing, and I couldn't cause a SOLR error after this, even while cranking it up to about 60+ requests per second. Another thing to note was that during the constant error storm, our CPU usage was hovering at about 95%, now its dropped to a sane level at about 10%. If this works for everyone, I'll commit it to trunk. Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Mon, May 10, 2010 at 5:30 AM, TAYLOR Robin robin.tay...@ed.ac.uk wrote: Hi Bill, I have found JMX/jconsolve invaluable when investigating memory problems http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html. Cheers, Robin. Robin Taylor Main Library University of Edinburgh Tel. 0131 6513808 -Original Message- From: Anderson, Charles W [mailto:bill.ander...@library.gatech.edu] Sent: 06 May 2010 16:41 To: dspace-tech Subject: [Dspace-tech] solr crashing dspace 1.6 We're on our second attempt to upgrade from 1.5.2 to 1.6. We have been plagued with m memory errors both times, and we' think we've narrowed it down to something that's going on in the solr webapp. We've got solr running in it's very own Tomcat instance; which has helped, but hasn't completely eliminated the problems. Just before we crash, we're seeing lots and lots of the following in the dspace log: org.apache.solr.common.SolrException: Internal Server Error Then, eventually, we start getting this: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error: Timeout waiting for idle object Meanwhile, over in catalina.out, we have these, which seem to correspond to the solr errors in the dspace log: SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. And, eventually, the voice of doom: Exception in thread TP-Processor550
Re: [Dspace-tech] solr crashing dspace 1.6
Hi Bill, I have found JMX/jconsolve invaluable when investigating memory problems http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html. Cheers, Robin. Robin Taylor Main Library University of Edinburgh Tel. 0131 6513808 -Original Message- From: Anderson, Charles W [mailto:bill.ander...@library.gatech.edu] Sent: 06 May 2010 16:41 To: dspace-tech Subject: [Dspace-tech] solr crashing dspace 1.6 We're on our second attempt to upgrade from 1.5.2 to 1.6. We have been plagued with m memory errors both times, and we' think we've narrowed it down to something that's going on in the solr webapp. We've got solr running in it's very own Tomcat instance; which has helped, but hasn't completely eliminated the problems. Just before we crash, we're seeing lots and lots of the following in the dspace log: org.apache.solr.common.SolrException: Internal Server Error Then, eventually, we start getting this: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error: Timeout waiting for idle object Meanwhile, over in catalina.out, we have these, which seem to correspond to the solr errors in the dspace log: SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. And, eventually, the voice of doom: Exception in thread TP-Processor550 java.lang.OutOfMemoryError: PermGen space Has anybody else experience memory issues related to solr? Thanks, Bill Bill Anderson Software Developer Digital Library Development Georgia Tech Library -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] solr crashing dspace 1.6
With the upgrade you essentially have another webapp running, i.e. SOLR, which means your instance will need more memory. More for when doing a lot of work, and more to be permanently used i.e permgen. When I've performed large batch processing (converting and importing old log files), I've bumped up my max memory for tomcat/java and increased the permgen. I'm not sure if there is something triggering solr to jump over the edge in your case, but after I've allocated more memory to java/tomcat, problems just go away. Thus for me it would be: edit /etc/init.d/tomcat6 JAVA_OPTS=-Djava.awt.headless=true -Xmx1536M *-XX:MaxPermSize=128m* or, however you normally set JAVA_OPTS Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Thu, May 6, 2010 at 11:41 AM, Anderson, Charles W bill.ander...@library.gatech.edu wrote: We're on our second attempt to upgrade from 1.5.2 to 1.6. We have been plagued with m memory errors both times, and we' think we've narrowed it down to something that's going on in the solr webapp. We've got solr running in it's very own Tomcat instance; which has helped, but hasn't completely eliminated the problems. Just before we crash, we're seeing lots and lots of the following in the dspace log: org.apache.solr.common.SolrException: Internal Server Error Then, eventually, we start getting this: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error: Timeout waiting for idle object Meanwhile, over in catalina.out, we have these, which seem to correspond to the solr errors in the dspace log: SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. And, eventually, the voice of doom: Exception in thread TP-Processor550 java.lang.OutOfMemoryError: PermGen space Has anybody else experience memory issues related to solr? Thanks, Bill Bill Anderson Software Developer Digital Library Development Georgia Tech Library -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] solr crashing dspace 1.6
I use: http://psi-probe.googlecode.com/ to monitor memory usage. Still have to get all the bells and whistles going thought. If anybody knows of a better open source app, please let me know. On 6 May 2010 19:27, Peter Dietz dietz...@osu.edu wrote: With the upgrade you essentially have another webapp running, i.e. SOLR, which means your instance will need more memory. More for when doing a lot of work, and more to be permanently used i.e permgen. When I've performed large batch processing (converting and importing old log files), I've bumped up my max memory for tomcat/java and increased the permgen. I'm not sure if there is something triggering solr to jump over the edge in your case, but after I've allocated more memory to java/tomcat, problems just go away. Thus for me it would be: edit /etc/init.d/tomcat6 JAVA_OPTS=-Djava.awt.headless=true -Xmx1536M *-XX:MaxPermSize=128m* or, however you normally set JAVA_OPTS Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Thu, May 6, 2010 at 11:41 AM, Anderson, Charles W bill.ander...@library.gatech.edu wrote: We're on our second attempt to upgrade from 1.5.2 to 1.6. We have been plagued with m memory errors both times, and we' think we've narrowed it down to something that's going on in the solr webapp. We've got solr running in it's very own Tomcat instance; which has helped, but hasn't completely eliminated the problems. Just before we crash, we're seeing lots and lots of the following in the dspace log: org.apache.solr.common.SolrException: Internal Server Error Then, eventually, we start getting this: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error: Timeout waiting for idle object Meanwhile, over in catalina.out, we have these, which seem to correspond to the solr errors in the dspace log: SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. And, eventually, the voice of doom: Exception in thread TP-Processor550 java.lang.OutOfMemoryError: PermGen space Has anybody else experience memory issues related to solr? Thanks, Bill Bill Anderson Software Developer Digital Library Development Georgia Tech Library -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Hilton Gibson Systems Administrator JSG Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Cell: +27 846 464 758 Email: hgibson__AT__sun.ac.za -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
