Re: [dspace-tech] DSpace Java OutOfMemory Errors

[Andrea Schweer]

Hi Chelsea, On 30/09/15 09:13, clobd...@swarthmore.edu wrote: We've been seeing DSpace crash a lot recently. Digging into the Catalina logs, I am seeing severe errors indicating a java memory leak: SEVERE: A web application registered the JBDC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. SEVERE: A web application created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Prototype beans currently in creation]) and a value of type [null] (value [null]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. SEVERE: A web application registered the JBDC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. SEVERE: A web application appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak. SEVERE: A web application appears to have started a thread named [TP-Processor6] but has failed to stop it. This is very likely to create a memory leak. SEVERE: A web application created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Prototype beans currently in creation]) and a value of type [null] (value [null]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. I think this is unrelated to the message below. These messages (unfortunately) are fairly typical when tomcat is shut down. Do you see anything in the catalina log from the time Tomcat crashed? Looking at the dspace logs, I am seeing many Java OutOfMemory errors: 2015-09-27 02:02:20,235 ERROR org.dspace.app.mediafilter.PDFFilter @ Error parsing PDF document Java heap space java.lang.OutOfMemoryError: Java heap space at org.apache.pdfbox.io.RandomAccessBuffer.expandBuffer(RandomAccessBuffer.java:151) at org.apache.pdfbox.io.RandomAccessBuffer.write(RandomAccessBuffer.java:131) at org.apache.pdfbox.io.RandomAccessFileOutputStream.write(RandomAccessFileOutputStream.java:108) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) at org.apache.pdfbox.pdmodel.encryption.SecurityHandler.encryptData(SecurityHandler.java:294) at org.apache.pdfbox.pdmodel.encryption.SecurityHandler.decryptStream(SecurityHandler.java:391) at org.apache.pdfbox.pdmodel.encryption.SecurityHandler.decrypt(SecurityHandler.java:363) at org.apache.pdfbox.pdmodel.encryption.SecurityHandler.decryptObject(SecurityHandler.java:337) at org.apache.pdfbox.pdmodel.encryption.SecurityHandler.proceedDecryption(SecurityHandler.java:177) at org.apache.pdfbox.pdmodel.encryption.StandardSecurityHandler.decryptDocument(StandardSecurityHandler.java:257) at org.apache.pdfbox.pdmodel.PDDocument.openProtection(PDDocument.java:1325) at org.apache.pdfbox.pdmodel.PDDocument.decrypt(PDDocument.java:796) at org.apache.pdfbox.util.PDFTextStripper.writeText(PDFTextStripper.java:310) at org.dspace.app.mediafilter.PDFFilter.getDestinationStream(PDFFilter.java:101) at org.dspace.app.mediafilter.MediaFilterManager.processBitstream(MediaFilterManager.java:737) at org.dspace.app.mediafilter.MediaFilterManager.filterBitstream(MediaFilterManager.java:561) at org.dspace.app.mediafilter.MediaFilterManager.filterItem(MediaFilterManager.java:511) at org.dspace.app.mediafilter.MediaFilterManager.applyFiltersItem(MediaFilterManager.java:479) at org.dspace.app.mediafilter.MediaFilterManager.applyFiltersAllItems(MediaFilterManager.java:414) at org.dspace.app.mediafilter.MediaFilterManager.main(MediaFilterManager.java:333) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) This message is coming from the media filter cron job, which isn't running under Tomcat at all. Our Java memory options are set to 4096M, which is half of the memory we have (8G ram): # dspace requires tomcat to use UTF-8, and also recommends memory parameters JAVA_OPTS="${JAVA_OPTS} -Xmx4096M -Xms4096M -Dfile.encoding=UTF-8" I'm assuming these are the JAVA_OPTS for tomcat? For the media filter, you'll also need to change Xms / Xms in [dspace]/bin/dspace -- that's where the command-line tools get their settings from. You're not saying which DSpace version you're on; in DSpace 5.x the setting is here: https://github.com/DSpace/DSpace/blob/dspace-5_x/dspace/bin/dspace#L77 and defaults to 256MB heap memory for the command-line tools. Actually, are you sure Tomcat is using the JAVA_OPTS you gave above? Could you have a look at the Java Information tab in your Control Panel (assuming XMLUI) and verify that the values under "Runtime statistics" look right (maximum memory should correspond to the 4GB you set in the JAVA_OPTS). We've just set up a cron to do nightly restarts of tomcat. I was just wondering if anyone else had some advice on handling these out of memory errors? Is there anything else we can do aside from the nightly restart? I think we'll need more information from the actual time Tomcat crashes. If you can't find anything, you might like to consider telling Tomcat to write out a heap dump when it runs out of memory that can then be analysed to figure out what's using all the memory: JAVA_OPTS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dir' The eclipse memory analyser tools is quite good for looking at the files produced this way, but of course you will need to have your tomcat crash at least once with that setting on! cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand +64-7-837 9120 -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at http://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.