https://bz.apache.org/bugzilla/show_bug.cgi?id=57860
Bug ID: 57860 Summary: use mina with apr/tomcat native and it crashs Product: Tomcat Native Version: 1.1.33 Hardware: PC OS: Linux Status: NEW Severity: critical Priority: P2 Component: Library Assignee: dev@tomcat.apache.org Reporter: zhangwei_e...@163.com Created attachment 32687 --> https://bz.apache.org/bugzilla/attachment.cgi?id=32687&action=edit java crash log I use mina with apr socket (AprSocketAcceptor and AprSocketConnector) for better perfermonce. the situation is simple: 1000 client connnet to mina server and keep alive connectons(long connection), and another server tell the mina server to push group messages every one minitue. After servel group push, the mina server not work and crashs. i just download and compile apr-1.5.1.tar.gz and tomcat-native-1.1.33, and modify the mina server code like this: if(useAPR){ SimpleIoProcessorPool<AprSession> pool = new SimpleIoProcessorPool<AprSession>(AprIoProcessor.class); acceptor = new AprSocketAcceptor(executor, pool); }else{ SimpleIoProcessorPool<NioSession> pool = new SimpleIoProcessorPool<NioSession>(NioProcessor.class); acceptor = new NioSocketAcceptor(executor, pool); } if(TcpServer.useAPR){ connector = new AprSocketConnector(); }else{ connector = new NioSocketConnector(); } Stack: [0x00007f5a8d327000,0x00007f5a8d3a8000], sp=0x00007f5a8d3a6470, free space=509k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libtcnative-1.so.0.1.33+0x148cf] Java_org_apache_tomcat_jni_Poll_poll+0x7f Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J org.apache.tomcat.jni.Poll.poll(JJ[JZ)I J org.apache.mina.transport.socket.apr.AprIoProcessor.select(J)I j org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run()V+52 j org.apache.mina.util.NamePreservingRunnable.run()V+29 j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Use gdb to [root@iZ25kxtgwarZ push_heartserver8888]# gdb /opt/jdk1.7.0_45/bin/java core.17249 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /opt/jdk1.7.0_45/bin/java...Missing separate debuginfo for /opt/jdk1.7.0_45/bin/java Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b8/2a586842f6da3df8a61093daadf04180e85c16.debug (no debugging symbols found)...done. [New Thread 17292] [New Thread 17312] [New Thread 17303] [New Thread 17294] [New Thread 17252] [New Thread 17307] [New Thread 17313] [New Thread 17304] [New Thread 17260] [New Thread 17255] [New Thread 17320] [New Thread 17251] [New Thread 17277] [New Thread 17325] [New Thread 17265] [New Thread 17249] [New Thread 17285] [New Thread 17295] [New Thread 17344] [New Thread 17343] [New Thread 17266] [New Thread 17256] [New Thread 17250] [New Thread 17290] [New Thread 17349] [New Thread 17253] [New Thread 17263] [New Thread 17262] [New Thread 17346] [New Thread 17258] [New Thread 17254] [New Thread 17272] [New Thread 17267] [New Thread 17347] [New Thread 17342] [New Thread 17287] [New Thread 17268] [New Thread 17257] [New Thread 17271] [New Thread 17286] [New Thread 17273] [New Thread 17297] [New Thread 17278] [New Thread 17274] [New Thread 17264] [New Thread 17348] [New Thread 17261] [New Thread 17259] [New Thread 17288] [New Thread 17308] [New Thread 17282] [New Thread 17345] Missing separate debuginfo for /opt/jdk1.7.0_45/bin/../lib/amd64/jli/libjli.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/34/5b529c72e59f506bf391be9a64cbbf7574211f Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/31a11f5f50e47d14c9871057d41df781aa6ccd Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libverify.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/20/b03b77b651a0152ab8de47774328f0cef51987 Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libjava.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/44/d3e7e98d5c21945a85d4ef6d5e6991a414ae6d Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libzip.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/07/c3e3ac6dc32953a6b623b6f9631d3ccc148800 Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libnio.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/26/2c5a10aaf4e557c40616d0f8d2988fb9d0d323 Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libnet.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a5/ce95e2069f89c08f90a4876a18acf6a6fe28c2 Missing separate debuginfo for /usr/local/apr/lib/libtcnative-1.so.0.1.33 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/64/c7c3ea9b1da0105a4831820cf77ce54b36dd5d Missing separate debuginfo for /usr/local/apr/lib/libapr-1.so.0 Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9f/82e698e6063202d19523bc3bbe9c96473fc73a Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/1e/0a7d58f454926e2afb4797865d85801ed65ece Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /opt/jdk1.7.0_45/bin/../lib/amd64/jli/libjli.so...(no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/bin/../lib/amd64/jli/libjli.so Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so...Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/31a11f5f50e47d14c9871057d41df781aa6ccd.debug (no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/libverify.so...Missing separate debuginfo for /opt/jdk1.7.0_45/jre/lib/amd64/libverify.so Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/20/b03b77b651a0152ab8de47774328f0cef51987.debug (no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/libverify.so Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/libjava.so...(no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/libjava.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/libzip.so...(no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/libzip.so Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/libnio.so...(no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/libnio.so Reading symbols from /opt/jdk1.7.0_45/jre/lib/amd64/libnet.so...(no debugging symbols found)...done. Loaded symbols for /opt/jdk1.7.0_45/jre/lib/amd64/libnet.so Reading symbols from /usr/local/apr/lib/libtcnative-1.so.0.1.33...done. Loaded symbols for /usr/local/apr/lib/libtcnative-1.so.0.1.33 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /usr/local/apr/lib/libapr-1.so.0...done. Loaded symbols for /usr/local/apr/lib/libapr-1.so.0 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Core was generated by `/opt/jdk1.7.0_45/bin/java -server -Xms14436m -Xms14436m -Xss512k -XX:MaxDirectM'. Program terminated with signal 6, Aborted. #0 0x0000003712632925 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.2.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-10.el6_5.x86_64 openssl-1.0.1e-30.el6_6.7.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x0000003712632925 in raise () from /lib64/libc.so.6 #1 0x0000003712634105 in abort () from /lib64/libc.so.6 #2 0x00007f5aef40f155 in os::abort(bool) () from /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so #3 0x00007f5aef58e087 in VMError::report_and_die() () from /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so #4 0x00007f5aef413adf in JVM_handle_linux_signal () from /opt/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so #5 <signal handler called> #6 Java_org_apache_tomcat_jni_Poll_poll (e=0x7f5a6801e1e8, o=<value optimized out>, pollset=140028420158600, timeout=<value optimized out>, set=0x7f5a8d3a64f0, remove=0 '\000') at src/poll.c:310 #7 0x00007f5ae517f6b5 in ?? () #8 0x00007f5a8d3a6530 in ?? () #9 0x0000000000000000 in ?? () (gdb) the core file is too big(>500MB) to attach. if you wish, i can provide the file. and i tried to fix the null point error by adding checking code like this: vim src/poll.c TCN_IMPLEMENT_CALL(jint, Poll, poll)(TCN_STDARGS, jlong pollset, jlong timeout, jlongArray set, jboolean remove) .... /* Find the minimum timeout */ APR_RING_FOREACH(ep, &p->poll_ring, tcn_pfde_t, link) { apr_interval_time_t socket_timeout = 0; tcn_socket_t *s = (tcn_socket_t *)ep->fd.client_data; if(s==NULL){ continue; } if (s->timeout == TCN_NO_SOCKET_TIMEOUT) { socket_timeout = p->default_timeout; } else { socket_timeout = s->timeout; } it then works and don't crash, but the mina can't receive messages any more after some time(the time it should crashs and exits) -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org