Hi all,
We're using openssl within a multi-threaded Java-based server which supports multiple
concurrent non-blocking socket connections. And on our production stack, the server
is crashing with this "Unexpected Signal : 11" error every 2-3 weeks or so (I think
its happen a total of 4 times in the last 2 months).
Obviously, this is very bad news for us because all users connected to that server get
disconnected, lose whatever they were doing at the time, and have to login again. Any
information about what might cause this error, how it might be resolved, or even just
some insight/explanation/references-to Signal 11 errors in general would be greatly
appreciated.
I should also mention that the only other stack on which we've seen this error was
performance/load-testing. However, this, and other errors, disappeared when we
implemented the locking callback methods for libcrypto. Makes me think that this is a
multi-threaded problem. But then, why are we still getting this error (but none of
the other multi-threaded-related errors) after moving this fix to production?
The error logs from the jvm from two seperate crashes along with relevant system and
environment information are included below:
Tue Nov 18 15:02:44 2003: SS: An unexpected exception has been detected in native code
outside the VM.
Tue Nov 18 15:02:44 2003: SS: Unexpected Signal : 11 occurred at PC=0xEB0C1838
Tue Nov 18 15:02:44 2003: SS: Function=MD5_Init+0x8
Tue Nov 18 15:02:44 2003: SS: Library=/opt/local/ssl/lib/libcrypto.so.0.9.7
Tue Nov 18 15:02:44 2003: SS:
Tue Nov 18 15:02:44 2003: SS: Current Java thread:
Tue Nov 18 15:02:44 2003: SS: at comm.iris.SSLUtil.SSLRead(Native Method)
Tue Nov 18 15:02:44 2003: SS: at comm.iris.SSLSocketChannel.readImpl
<snip>
Tue Nov 18 15:02:44 2003: SS: Dynamic libraries:
Tue Nov 18 15:02:44 2003: SS: 0x10000 /usr/local/java/bin/java
Tue Nov 18 15:02:44 2003: SS: 0xff350000 /usr/lib/libthread.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff390000 /usr/lib/libdl.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff200000 /usr/lib/libc.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff340000
/usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1
Tue Nov 18 15:02:44 2003: SS: 0xfe000000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/server/libjvm.so
Tue Nov 18 15:02:44 2003: SS: 0xff2e0000 /usr/lib/libCrun.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff1e0000 /usr/lib/libsocket.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff100000 /usr/lib/libnsl.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff0d0000 /usr/lib/libm.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff310000 /usr/lib/libw.so.1
Tue Nov 18 15:02:44 2003: SS: 0xff0b0000 /usr/lib/libmp.so.2
Tue Nov 18 15:02:44 2003: SS: 0xff080000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/native_threads/libhpi.so
Tue Nov 18 15:02:44 2003: SS: 0xff050000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libverify.so
Tue Nov 18 15:02:44 2003: SS: 0xfe7c0000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libjava.so
Tue Nov 18 15:02:44 2003: SS: 0xff030000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libzip.so
Tue Nov 18 15:02:44 2003: SS: 0xfdf80000 /usr/lib/nss_files.so.1
Tue Nov 18 15:02:44 2003: SS: 0xfa1d0000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libnet.so
Tue Nov 18 15:02:44 2003: SS: 0xfa290000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libnio.so
Tue Nov 18 15:02:44 2003: SS: 0xfa1b0000 /usr/lib/librt.so.1
Tue Nov 18 15:02:44 2003: SS: 0xfa190000 /usr/lib/libaio.so.1
Tue Nov 18 15:02:44 2003: SS: 0xeb080000 /opt/local/ssl/lib/libcrypto.so.0.9.7
Tue Nov 18 15:02:44 2003: SS: 0xfa0e0000 /usr/local/lib/libgcc_s.so.1
Tue Nov 18 15:02:44 2003: SS: 0xfa090000 /opt/local/ssl/lib/libssl.so.0.9.7
Tue Nov 18 15:02:44 2003: SS: 0xfa070000
/server/cxserver.Blue_3.0.21589/lib/libiris.so
Tue Nov 18 15:02:44 2003: SS:
Tue Nov 18 15:02:44 2003: SS: Local Time = Tue Nov 18 15:02:44 2003
Tue Nov 18 15:02:44 2003: SS: Elapsed Time = 230525
Tue Nov 18 15:02:44 2003: SS: #
Tue Nov 18 15:02:44 2003: SS: # The exception above was detected in native code
outside the VM
Tue Nov 18 15:02:44 2003: SS: #
Tue Nov 18 15:02:44 2003: SS: # Java VM: Java HotSpot(TM) Server VM (1.4.1_02-b06
mixed mode)
Tue Nov 18 15:02:44 2003: SS: #
Tue Nov 18 15:02:44 2003: SS: # An error report file has been saved as
hs_err_pid29727.log.
Tue Nov 18 15:02:44 2003: SS: # Please refer to the file for further information.
Dec 9 16:07:39 SS: Unexpected Signal : 11 occurred at PC=0xEB21FD9C
Dec 9 16:07:39 181760 SS: Function=EVP_DigestUpdate+0xC
Dec 9 16:07:39 190885 SS: Library=/opt/local/ssl/lib/libcrypto.so.0.9.7
Dec 9 16:07:39 448413 SS:
Dec 9 16:07:39 909024 SS: Current Java thread:
Dec 9 16:07:39 770000 SS: at comm.iris.SSLUtil.SSLWrite(Native Method)
Dec 9 16:07:39 706525 SS: at
comm.iris.SSLSocketChannel.writeImpl(SSLSocketChannel.java:82)
<snip>
Dec 9 16:07:39 SS: Dynamic libraries:
Dec 9 16:07:39 SS: 0x10000 /usr/local/java/bin/java
Dec 9 16:07:39 SS: 0xff350000 /usr/lib/libthread.so.1
Dec 9 16:07:39 SS: 0xff390000 /usr/lib/libdl.so.1
Dec 9 16:07:39 SS: 0xff200000 /usr/lib/libc.so.1
Dec 9 16:07:39 SS: 0xff340000 /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1
Dec 9 16:07:39 SS: 0xfe000000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/server/libjvm.so
Dec 9 16:07:39 SS: 0xff2e0000 /usr/lib/libCrun.so.1
Dec 9 16:07:39 SS: 0xff1e0000 /usr/lib/libsocket.so.1
Dec 9 16:07:39 SS: 0xff100000 /usr/lib/libnsl.so.1
Dec 9 16:07:39 SS: 0xff0d0000 /usr/lib/libm.so.1
Dec 9 16:07:39 SS: 0xff310000 /usr/lib/libw.so.1
Dec 9 16:07:39 SS: 0xff0b0000 /usr/lib/libmp.so.2
Dec 9 16:07:39 SS: 0xff080000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/native_threads/libhpi.so
Dec 9 16:07:39 SS: 0xff050000
/opt/local/j2sdk1.4.1_02/jre/lib/sparc/libverify.so
Dec 9 16:07:39 SS: 0xfe7c0000 /opt/local/j2sdk1.4.1_02/jre/lib/sparc/libjava.so
Dec 9 16:07:39 SS: 0xff030000 /opt/local/j2sdk1.4.1_02/jre/lib/sparc/libzip.so
Dec 9 16:07:39 SS: 0xfdf80000 /usr/lib/nss_files.so.1
Dec 9 16:07:39 SS: 0xfa1d0000 /opt/local/j2sdk1.4.1_02/jre/lib/sparc/libnet.so
Dec 9 16:07:39 SS: 0xfa290000 /opt/local/j2sdk1.4.1_02/jre/lib/sparc/libnio.so
Dec 9 16:07:39 SS: 0xfa1b0000 /usr/lib/librt.so.1
Dec 9 16:07:39 SS: 0xfa190000 /usr/lib/libaio.so.1
Dec 9 16:07:39 SS: 0xeb180000 /opt/local/ssl/lib/libcrypto.so.0.9.7
Dec 9 16:07:39 SS: 0xfa0e0000 /usr/local/lib/libgcc_s.so.1
Dec 9 16:07:39 SS: 0xfa090000 /opt/local/ssl/lib/libssl.so.0.9.7
Dec 9 16:07:39 SS: 0xfa070000 /server/cxserver.Blue_3.0.21589/lib/libiris.so
Dec 9 16:07:39 SS:
Dec 9 16:07:39 SS: Local Time = Tue Dec 9 16:07:39 2003
Dec 9 16:07:39 SS: Elapsed Time = 234447
Dec 9 16:07:39 SS: #
Dec 9 16:07:39 SS: # The exception above was detected in native code outside the VM
Dec 9 16:07:39 SS: #
Dec 9 16:07:39 SS: # Java VM: Java HotSpot(TM) Server VM (1.4.1_02-b06 mixed mode)
Dec 9 16:07:39 SS: #
Dec 9 16:07:39 SS: # An error report file has been saved as hs_err_pid15462.log.
Dec 9 16:07:39 SS: # Please refer to the file for further information.
Information about the environment and hardware on which the server runs:
java version "1.4.1_02" (This is the 64b version)
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_02-b06)
Java HotSpot(TM) Client VM (build 1.4.1_02-b06, mixed mode)
Java args: -server -ms128m -mx128m -verbose:gc
OpenSSL 0.9.7b 10 Apr 2003
SunOS tango 5.8 Generic_108528-17 sun4u sparc SUNW,Sun-Fire-280R
System Configuration: Sun Microsystems sun4u Sun Fire 280R (2 X UltraSPARC-III)
System clock frequency: 150 MHz
Memory size: 2048 Megabytes
The code that sets up the locking callbacks required for multi-threaded use:
static mutex_t *lock_cs;
void locking_callback(int mode, int type, char *file, int line)
{
/*
fprintf(stderr,"thread=%4d mode=%s lock=%s %s:%d\n", CRYPTO_thread_id(),
(mode&CRYPTO_LOCK)?"l":"u", (type&CRYPTO_READ)?"r":"w", file, line);
*/
if (mode & CRYPTO_LOCK) mutex_lock(&(lock_cs[type]));
else mutex_unlock(&(lock_cs[type]));
}
unsigned long thread_id(void) { return (unsigned long) thr_self(); }
lock_cs = OPENSSL_malloc(CRYPTO_num_locks() * sizeof(mutex_t));
for (i = 0; i < CRYPTO_num_locks(); i++)
{
mutex_init(&(lock_cs[i]), USYNC_THREAD, NULL);
}
CRYPTO_set_id_callback((unsigned long (*)()) thread_id);
CRYPTO_set_locking_callback((void (*)()) locking_callback);
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
