Hi Graham!
Problem seems to be gone. Last occurrence was Apr 19th. messages.1:Apr 15 13:24:47 dspace kernel: java[24630]: segfault at 0000000043962788 rip 00002aaab2750360 rsp 0000000043962780 error 6 messages.1:Apr 15 16:37:53 dspace kernel: java[26841]: segfault at 0000000041763508 rip 00002aaab2750360 rsp 0000000041763500 error 6 messages.1:Apr 15 20:04:35 dspace kernel: java[4214]: segfault at 0000000041139938 rip 00002aaab2750360 rsp 0000000041139930 error 6 messages.1:Apr 15 21:49:40 dspace kernel: java[6530]: segfault at 00000000410c9608 rip 00002aaab2750360 rsp 00000000410c9600 error 6 messages.1:Apr 16 01:39:19 dspace kernel: java[15152]: segfault at 00000000447f98a8 rip 00002aaab274d360 rsp 00000000447f98a0 error 6 messages.1:Apr 16 17:02:21 dspace kernel: java[8687]: segfault at 0000000040f47658 rip 00002aaab2750360 rsp 0000000040f47650 error 6 messages.1:Apr 19 20:11:54 dspace kernel: java[15168]: segfault at 00000000444683a8 rip 00002aaab2750360 rsp 00000000444683a0 error 6 messages.2:Apr 12 19:17:08 dspace kernel: java[25162]: segfault at 0000000040cba848 rip 00002aaab237b360 rsp 0000000040cba840 error 6 messages.2:Apr 13 00:23:48 dspace kernel: java[31874]: segfault at 0000000040f8cd38 rip 00002aaab237b360 rsp 0000000040f8cd30 error 6 messages.2:Apr 13 00:59:13 dspace kernel: java[10759]: segfault at 0000000043ee8918 rip 00002aaab237b360 rsp 0000000043ee8910 error 6 messages.2:Apr 13 01:46:43 dspace kernel: java[12332]: segfault at 00000000416ad528 rip 00002aaab237b360 rsp 00000000416ad520 error 6 messages.2:Apr 13 03:21:06 dspace kernel: java[13730]: segfault at 00000000443ca9a8 rip 00002aaab23fb360 rsp 00000000443ca9a0 error 6 messages.2:Apr 13 13:28:17 dspace kernel: java[26282]: segfault at 00000000449fc5c8 rip 00002aaab23fb360 rsp 00000000449fc5c0 error 6 messages.2:Apr 13 15:04:50 dspace kernel: java[1841]: segfault at 0000000043c87698 rip 00002aaab23fb360 rsp 0000000043c87690 error 6 messages.2:Apr 14 01:35:42 dspace kernel: java[2947]: segfault at 000000004414f5e8 rip 00002aaab23fb360 rsp 000000004414f5e0 error 6 messages.2:Apr 15 01:14:10 dspace kernel: java[21030]: segfault at 0000000044c68e28 rip 00002aaab23fb360 rsp 0000000044c68e20 error 6 messages.3:Apr 3 21:52:26 dspace kernel: java[26152]: segfault at 0000000043b86558 rip 00002aaab237b360 rsp 0000000043b86550 error 6 messages.3:Apr 4 00:10:34 dspace kernel: java[1269]: segfault at 0000000043c69548 rip 00002aaab237b360 rsp 0000000043c69540 error 6 messages.3:Apr 4 13:30:16 dspace kernel: java[32300]: segfault at 0000000043c83808 rip 00002aaab237b360 rsp 0000000043c83800 error 6 messages.3:Apr 4 14:38:22 dspace kernel: java[6215]: segfault at 00000000440af908 rip 00002aaab237b360 rsp 00000000440af900 error 6 messages.3:Apr 6 01:41:40 dspace kernel: java[14157]: segfault at 0000000044922608 rip 00002aaab237b360 rsp 0000000044922600 error 6 messages.3:Apr 6 03:16:59 dspace kernel: java[16931]: segfault at 000000004392f668 rip 00002aaab2e0a360 rsp 000000004392f660 error 6 messages.3:Apr 6 07:23:18 dspace kernel: java[29903]: segfault at 0000000041ea3778 rip 00002aaab237b360 rsp 0000000041ea3770 error 6 messages.3:Apr 6 21:23:01 dspace kernel: java[24050]: segfault at 0000000043b0f5a8 rip 00002aaab237b360 rsp 0000000043b0f5a0 error 6 messages.3:Apr 7 12:45:45 dspace kernel: java[24703]: segfault at 00000000436ed778 rip 00002aaab237b360 rsp 00000000436ed770 error 6 messages.4:Mar 28 09:31:44 dspace kernel: java[5061]: segfault at 000000004186a5c8 rip 00002aaab237b360 rsp 000000004186a5c0 error 6 messages.4:Mar 29 10:41:48 dspace kernel: java[1733]: segfault at 00000000424715a8 rip 00002aaab237b360 rsp 00000000424715a0 error 6 messages.4:Mar 29 21:22:57 dspace kernel: java[26636]: segfault at 000000004358e608 rip 00002aaab237b360 rsp 000000004358e600 error 6 messages.4:Mar 30 16:29:07 dspace kernel: java[1784]: segfault at 0000000040e9b5a8 rip 00002aaab237b360 rsp 0000000040e9b5a0 error 6 messages.4:Mar 31 09:11:25 dspace kernel: java[6331]: segfault at 0000000043bdc618 rip 00002aaaaeda4360 rsp 0000000043bdc610 error 6 It can be a mixture of several things. Also seems to be related to some other errors that where arising ERROR, WARNING like WARN org.dspace.app.webui.util.UIUtil @ No context, the database might be down or the connection pool exhausted. Or Caused by: org.postgresql.util.PSQLException: This statement has been closed. But till now my actions to solve the problem concerning the connection pool exhausted and the statement has been closed seem to have a positive result on segfault error 6. You can think I am shooting on the dark but with black box testing the headache seems to be gone. 1. Increase number of connections to DB on postgres and DSpace 2. Increase Mx on tomcat 3. Free cache memory every hour with a cron script 4. Finally Changing db.maxidle from -1 to 20 Changing db.statementpool from true to false This fourth change seems to be the definitive one. A collateral effect is latency. Dspace takes a little more to render results but the positive point is stability has arrived. Bram Luyten give some hints https://jira.duraspace.org/browse/DS-320 though our DSpace instance is JSPUI and should be fixed because is 1.6.2 version. I havent test on source code if patches they say are already made, I suppose they are. All log files have now innocent lines except this one on tomcat Yesterday there were 10846 lines on tomcat logs with messages of this kind INFO: [statistics] PERFORMANCE WARNING: Overlapping onDeckSearchers= Today, just 73. Regards Evelio Martínez De: Graham Triggs [mailto:[email protected]] Enviado el: lunes, 23 de abril de 2012 10:59 Para: Evelio Martínez CC: [email protected] Asunto: Re: [Dspace-tech] dspace kernel java segfault error 6 Hi Evelio, It's very rare to see segfaults from Java in normal circumstances. And it's highly unlikely that the Apache proxy has any effect on it. There are really only two likely culprits for a segfault in this case: a) A bug in the Java virtual machine - this could be more likely if you are running a JVM other than Oracle / Sun (e.g. OpenJDK, particularly before 7). b) Any native code executed from Java via JNI. Given that you are using Postgres, the only likely route for native code being used with your Tomcat instance is if it is using the Apache Portable Runtime (APR). Whilst this can make the incoming connections more efficient, I've personally had nothing but grief from attempting to use APR with Tomcat. Regards, G On 15 April 2012 21:19, Evelio Martínez <[email protected]> wrote: Hi all! We have a DSpace instance with nearly 17000 items Components/version: - RHEL 5.5 (6 Gb RAM) - Postgresql 8.4.2 max_connections=500 - DSpace 1.6.2 db.maxconnections = 100 - Tomcat 6.0.33 with mod_jk 1.2.32 JAVA_OPTS="$JAVA_OPTS -Xmx2048m -Xms512m -XX:MaxPermSize=384m - Apache MPM Worker 2.2.21 - JDK 1.6.0_30 We are having Tomcat stopped suddenly with these messages: Apr 15 13:24:47 dspace kernel: java[24630]: segfault at 0000000043962788 rip 00002aaab2750360 rsp 0000000043962780 error 6 Apr 15 16:37:53 dspace kernel: java[26841]: segfault at 0000000041763508 rip 00002aaab2750360 rsp 0000000041763500 error 6 Apr 15 20:04:35 dspace kernel: java[4214]: segfault at 0000000041139938 rip 00002aaab2750360 rsp 0000000041139930 error 6 Any suggestions to debug the problem? Thanks in advance cid:f0154016-8e66-4c6b-9255-4e7524133341 Evelio Martínez Director proyectos software. CISA, CCNA Dept. de Tecnologías y Sistemas de Información C/ Garcilaso, 15-bajo 46003 Valencia Telf: 96 369 41 23 / 677 405 909 Fax: 96 369 34 39 <http://www.masmedios.com/> www.masmedios.com cid:f0154016-8e66-4c6b-9255-4e7524133341 cid:65691c43-2689-4f20-9b5d-5425927026a4 Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, puede contener información de carácter confidencial exclusivamente dirigida a su(s) destinatario(s). De acuerdo con la LOPD, MASmedios informa de que los datos personales contenidos quedarán sometidos a un tratamiento del que aquélla es responsable. Los interesados consienten el tratamiento de sus datos para estos fines, pudiendo ejercitar sus derechos de acceso, rectificación, cancelación y oposición dirigiéndose por escrito a MASmedios para la gestión de la información, calle Garcilaso 15-B 46003 Valencia o en <mailto:[email protected]> [email protected] ---------------------------------------------------------------------------- -- For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
<<image001.png>>
<<image002.gif>>
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
