Paul, It sounds like you are running into a per-user file handle limit. I've seen this with other application running on RHEL4 servers. It looks like your system-wide limit is okay, which is the limit applied to the root user. All other users are limited to 1024 files by default.
Since tomcat is probably running as a different user, you'll need to increase the total number of files that this user can have open at one time. Keep in mind that to *nix, everything is considered a file, including directories, libraries, etc. To allow another user to have more than 1024 files open add a line like this to /etc/security/limits.conf: tomcat - nofile 4096 You can change the user (including *) and number of files as appropriate. Login again (not via su) and check to see if the change worked: ulimit -n This should report 4096 rather than the 1024 default. If you still see 1024, try upgrading sshd since I think older versions don't refresh this var correctly. If you need more info, RedHat has a FAQ about all of this: http://kbase.redhat.com/faq/FAQ_80_1540.shtm I hope this helps, Eric Luhrs Digital Initiatives Librarian Lafayette College On Fri, Feb 15, 2008 at 3:46 PM, Paul Marlow <[EMAIL PROTECTED]> wrote: > Hello! > > First of all, forgive me as I'm new to DSpace, nor am I the one who > set up the system. > > That being said, we are consistently receiving errors referring to > "Too many files open", in which we must restart the Tomcat service > two or more times a day (which has been happening for nearly a month > now). Searching on our system is fine, it occurs when our users are > attempting to edit the metadata and/or uploading/removing bitstreams > - or when the system attempts to send email alerts. At this point, > other than searching, the system is unusable until the services are > restarted. > > I have done several searches in Google as well as in the Dspace > mailing lists, but I haven't found an actual solution to the problem, > although I have found discussions on the topic. > > In relation to all the discussions I've read, I have provided most of > the information requested by them in a hope that others may have an > answer, or at least speed things up a bit. > > DSpace version: > 1.4.1 > > Operating System: > OS Version: > # cat /etc/redhat-release > Red Hat Enterprise Linux AS release 4 (Nahant Update 4) > # uname -a > Linux <node name removed> 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 > 17:17:21 EST 2007 i686 i686 i386 GNU/Linux > > Cron Jobs: > dspace cron jobs > ]# crontab -u dspace -l > # Send out subscription emails at 1:00 everyday > 0 1 * * * /opt/dspace/bin/sub-daily > #Run the media filter at 2:00 every day > 0 2 * * * /opt/dspace/bin/filter-media > #Run the checksum checker at 3:00 > 0 3 * * * /opt/dspace/bin/checker -lp > #Mail the results to the sysadmin at 4:00 > 0 4 * * * /opt/dspace/bin/dsrun org.dspace.checker.DailyReportEmailer > -c > #Run stat analyses > 0 5 * * * /opt/dspace/bin/stat-general > 10 5 * * * /opt/dspace/bin/stat-monthly > 20 5 * * * /opt/dspace/bin/stat-report-general > 30 5 * * * /opt/dspace/bin/stat-report-monthly > > Notice that index-all is not included, as my understand is that > filter-media includes a rebuild of the indexes. I have manually > rebuilt the indexes which does not resolve the issue. > > INodes: > # df -i > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/cciss/c0d0p1 524288 111084 413204 22% / > none 214062 1 214061 1% /dev/shm > /dev/cciss/c0d0p3 17137664 644078 16493586 4% /var > > files-nr Information: > # cat /proc/sys/fs/file-nr > 5120 0 580170 > > Example Error from Tomcat's catalina.out: > Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > SEVERE: Endpoint > ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored > exception: java.net.SocketException: Too many open files > java.net.SocketException: Too many open files > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) > at java.net.ServerSocket.implAccept(ServerSocket.java:448) > at java.net.ServerSocket.accept(ServerSocket.java:419) > at > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Def > aultServerSocketFactory.java:61) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoin > t.java:408) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll > owerWorkerThread.java:71) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo > ol.java:685) > at java.lang.Thread.run(Thread.java:534) > Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > > Example corresponding errors in dspace.log: > java.io.FileNotFoundException: > /var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116 > 0566 (Too many open files) > va.io.FileNotFoundException: > /var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116 > 0566 (Too many open files) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:106) > at > edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream. > java:171) > at > edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream. > java:145) > at > edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStrea > m.java:139) > at > edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630) > at > org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(Bitstream > StorageManager.java:553) > at org.dspace.content.Bitstream.retrieve(Bitstream.java:512) > at > org.dspace.app.webui.servlet.BitstreamServlet.doDSGet(BitstreamServlet > .java:205) > at > org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServle > t.java:159) > at > org.dspace.app.webui.servlet.DSpaceServlet.doGet(DSpaceServlet.java:10 > 0) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lve.java:210) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > lve.java:174) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > va:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > va:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv > e.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > :151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java: > 870) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr > ocessConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi > nt.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll > owerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo > ol.java:685) > at java.lang.Thread.run(Thread.java:534) > 2008-02-15 10:28:06,006 WARN org.dspace.app.webui.util.UIUtil @ > Unable to send email alert > java.io.FileNotFoundException: > /opt/dspace/config/emails/internal_error (Too many open files) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:106) > at java.io.FileInputStream.<init>(FileInputStream.java:66) > at java.io.FileReader.<init>(FileReader.java:41) > at > org.dspace.core.ConfigurationManager.getEmail(ConfigurationManager.jav > a:282) > at org.dspace.app.webui.util.UIUtil.sendAlert(UIUtil.java:533) > at > org.dspace.app.webui.servlet.InternalErrorServlet.doGet(InternalErrorS > ervlet.java:86) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:188) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispa > tcher.java:691) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicat > ionDispatcher.java:471) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDi > spatcher.java:403) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDisp > atcher.java:301) > at > org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.ja > va:364) > at > org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve > .java:213) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > va:135) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > va:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv > e.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > :151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java: > 870) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr > ocessConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi > nt.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll > owerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo > ol.java:685) > at java.lang.Thread.run(Thread.java:534) > > > Any assistance would be greatly appreciated. > > Thanks! > > Paul Marlow > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
