Am 21.01.2019 um 15:49 schrieb Mark Thomas:
Version 1.2.20 includes the following changes compared to 1.2.19:
- Fixed memory leak associated with the use of OpenSSL BIO (used when
OpenSSL provides the TLS functionality for the NIO and NIO2
connectors)
Various other fixes and improvements. See the changelog for details.
The proposed release artefacts can be found at [1],
and the build was done using tag [2].
The Apache Tomcat Native 1.2.20 release is
[ ] Stable, go ahead and release
[ ] Broken because of ...
I have a strange finding on Solaris running the TC 8.5 head test suite
using TC Native 1.2.20, APR 1.6.5 and OpenSSL 1.0.2q, Java 1.8.0_201-b09:
I get crashes in TestCustomSsl for NIO and NIO2.
Test logs say
Testsuite: org.apache.tomcat.util.net.TestCustomSsl
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
Testcase: testCustomSslImplementation took 0.011 sec
Caused an ERROR
Forked Java VM exited abnormally. Please note the time in the report
does not reflect the time until the VM exit.
junit.framework.AssertionFailedError: Forked Java VM exited
abnormally. Please note the time in the report does not reflect the
time until the VM exit.
tcnative start logs:
Loaded APR based Apache Tomcat Native library [1.2.20] using APR
version [1.6.5].
APR capabilities: IPv6 [true], sendfile [true], accept filters
[false], random [true].
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
OpenSSL successfully initialized [OpenSSL 1.0.2q 20 Nov 2018]
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0xfffffffea171dda0, pid=8716,
tid=0x0000000000000007
#
# JRE version: Java(TM) SE Runtime Environment (8.0_201-b09) (build
1.8.0_201-b09)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# C [libapr-1.so.0.6.5+0x1dda0] apr_pool_destroy+0x10
#
# Core dump written. Default location: /path/to/core or core.8716
#
# An error report file with more information is saved as:
# /path/to/hs_err_pid8716.log
[thread 57 also had an error]
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
SIGBUS on Solaris typically means an alignment error, x86 is more
tolerant in this respect.
From hs_err.pid:
Current thread (0x00000001002d1000): JavaThread "Finalizer" daemon
[_thread_in_native, id=7, stack(0xfffffffea4000000,0xfffffffea4100000)]
siginfo: si_signo: 10 (SIGBUS), si_code: 1 (BUS_ADRALN), si_addr:
0x0001000b6c4e07f5
...
O0=0x0000000100f06af0 is an unknown value
O1=0x0000000000000000 is an unknown value
O2=0xffffffff7f048240: _uberdata+0 in /lib/sparcv9/libc.so.1 at
0xffffffff7ee00000
O3=0xffffffff7f04354c: __malloc_lock+0x1c in /lib/sparcv9/libc.so.1 at
0xffffffff7ee00000
O4=0xffffffff7f303200 is an unknown value
O5=0xfffffffea1bec768: OPENSSL_NONPIC_relocated+0x60 in
/path/to/openssl-1.0.2q-1-64.solaris10.sparc/lib/libcrypto.so.1.0.0 at
0xfffffffea1900000
O6=0xfffffffea40fe491 is pointing into the stack for thread:
0x00000001002d1000
O7=0xfffffffea1e1de3c: Java_org_apache_tomcat_jni_SSL_freeSSL+0x38 in
/path/to/native/.libs/libtcnative-1.so.0.2.20 at 0xfffffffea1e00000
...
Stack: [0xfffffffea4000000,0xfffffffea4100000],
sp=0xfffffffea40fec90, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
C [libapr-1.so.0.6.5+0x1dda0] apr_pool_destroy+0x10
j org.apache.tomcat.jni.SSL.freeSSL(J)V+0
j org.apache.tomcat.jni.SSL.freeSSL(J)V+0
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.shutdown()V+23
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.finalize()V+5
J 1268 C1
java.lang.ref.Finalizer.runFinalizer(Lsun/misc/JavaLangAccess;)V (62
bytes) @ 0xffffffff6c98259c [0xffffffff6c981ba0+0x9fc]
J 1267 C1
java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;Lsun/misc/JavaLangAccess;)V
(6 bytes) @ 0xffffffff6c981778 [0xffffffff6c9816c0+0xb8]
j java.lang.ref.Finalizer$FinalizerThread.run()V+45
v ~StubRoutines::call_stub
V [libjvm.so+0x6a0c8c] void
JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x94c
V [libjvm.so+0x69f23c] void
JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x31c
V [libjvm.so+0x69f4f8] void
JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x50
V [libjvm.so+0x7bc518] void thread_entry(JavaThread*,Thread*)+0xd8
V [libjvm.so+0xc168cc] void JavaThread::thread_main_inner()+0x94
V [libjvm.so+0xc16818] void JavaThread::run()+0x2b0
V [libjvm.so+0xa91f08] java_start+0x380
C [libc.so.1+0xd6eb8] _lwp_start+0x8
From the core dump:
ffffffff7eedaf1c _lwp_kill (6, 0, ffffffff7f047830,
ffffffffffffffff, ffffffff7f03c000, 0) + 8
ffffffff7ee4b310 abort (1, 1d8, ffffffff7e2a823c, 1f0dfc, 0, 0) + 118
ffffffff7e293e48 __1cCosFabort6Fb_v_ (1, 1, 3eba8, ffffffff7e793f90,
50019c, 3e800) + 58
ffffffff7e4c121c __1cHVMErrorOreport_and_die6M_v_ (1,
ffffffff7e80bffd, 1002d1000, ffffffff7e2a5190, ffffffff7e844e20,
ffffffff7e830c98) + 1094
ffffffff7e2a2ef8 JVM_handle_solaris_signal (a, fffffffea40feb90,
fffffffea171dda0, fffffffea40fe420, ffffffffffb07d80,
fffffffea40fe8b0) + b90
ffffffff7e29bd2c signalHandler (a, fffffffea40feb90,
fffffffea40fe8b0, ffffffff7f04cba0, 1002d1000, 0) + 1c
ffffffff7eed6fdc __sighndlr (a, fffffffea40feb90, fffffffea40fe8b0,
ffffffff7e29bd10, 0, 9) + c
ffffffff7eecab70 call_user_handler (ffffffff7f303200,
ffffffff7f303200, fffffffea40fe8b0, c, 0, 0) + 3e0
ffffffff7eecad7c sigacthandler (0, fffffffea40feb90,
fffffffea40fe8b0, ffffffff7f303200, 0, ffffffff7f03c000) + 68
--- called from signal handler with signal 0 (SIGEXIT) ---
fffffffea171dda0 apr_pool_destroy (10095f850, fffffffea40fef10,
100f06af0, 0, 100ea2a80, 100e45200) + 10
ffffffff6c411d50 * org/apache/tomcat/jni/SSL.freeSSL(J)V+0
ffffffff6c411cf4 * org/apache/tomcat/jni/SSL.freeSSL(J)V+0
ffffffff6c408068 *
org/apache/tomcat/util/net/openssl/OpenSSLEngine.shutdown()V+23
ffffffff6c408068 *
org/apache/tomcat/util/net/openssl/OpenSSLEngine.finalize()V+5
ffffffff6c982594 *
*java/lang/System$2.invokeFinalize(Ljava/lang/Object;)V [compiled] +2
ffffffff6c982594 *
*java/lang/ref/Finalizer.runFinalizer(Lsun/misc/JavaLangAccess;)V+47
ffffffff6c981770 *
java/lang/ref/Finalizer.access$100(Ljava/lang/ref/Finalizer;Lsun/misc/JavaLangAccess;)V+-136
ffffffff6c408068 * java/lang/ref/Finalizer$FinalizerThread.run()V+45
ffffffff6c40024c * StubRoutines (1)
ffffffff7dea0c84
__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_
(fffffffea40ffc60, 1, fffffffea40ffb08, ffffffff6c4001e0,
fffffffea54f8d68, 1002d1000) + 944
ffffffff7de9f234
__1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_
(fffffffea40ffc60, 1e, 8f3c00, 1002d1000, fffffffea40ffb08, 1002d1000)
+ 314
ffffffff7de9f4f0
__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_6pnGThread__v_
(100000000, 1002d1dc0, 8, 1, fffffffea40ffb51, fffffffea40ffb10) + 48
ffffffff7dfbc510 __1cMthread_entry6FpnKJavaThread_pnGThread__v_
(1002d1dc0, e, 100124ec0, 1002d1dc0, 7d7b4c, ffffffff7e837c28) + d0
ffffffff7e4168c4 __1cKJavaThreadRthread_main_inner6M_v_ (1002d1000,
3d8, 1001067c0, 1002d19b0, 0, 87400) + 8c
ffffffff7e416810 __1cKJavaThreadDrun6M_v_ (1002d1000, 7,
ffffffff41188f80, 1002d19c0, ffffffff7e793f90, 3) + 2a8
ffffffff7e291f00 java_start (1002d1000, ffffffff7e830418, 0,
1002bfb60, ffffffff7e793f90, c7be4) + 378
ffffffff7eed6eb0 _lwp_start (0, 0, 0, 0, 0, 0)
and strangely on an unrelated thread:
----------------- lwp# 57 / thread# 57 --------------------
ffffffff7eeda110 ___nanosleep (64, 0, 0, 0, ffffffff7f30ea00, 64) + 8
ffffffff7e299890 __1cCosOinfinite_sleep6F_v_ (ffffffff7e844e20, 8,
1, 0, 1, ffffffff7e60a60c) + 8
ffffffff7e4c032c __1cHVMErrorOreport_and_die6M_v_ (fffffffe9e0fcb20,
153800, 39, ffffffff7e830c98, ffffffff7e2a5190, ffffffff7e844e20) + 1a4
ffffffff7e4c1ac8 __1cNcrash_handler6FipnHsiginfo_pv_v_ (b, 0,
fffffffe9e0fcee0, ffffffff7f04cba0, fffffffe9e0fcdb0,
fffffffe9e0fcb80) + 48
ffffffff7eed6fdc __sighndlr (b, 0, fffffffe9e0fcee0,
ffffffff7e4c1a80, 0, a) + c
ffffffff7eecab70 call_user_handler (ffffffff7f30ea00,
ffffffff7f30ea00, fffffffe9e0fcee0, 4, 0, 0) + 3e0
ffffffff7eecad7c sigacthandler (0, 0, fffffffe9e0fcee0,
ffffffff7f30ea00, 0, ffffffff7f03c000) + 68
--- called from signal handler with signal 0 (SIGEXIT) ---
ffffffff6c7a89c0 *
*com/sun/org/apache/xerces/internal/impl/XMLEntityScanner.scanName(Lcom/sun/org/apache/xerces/internal/impl/XMLScanner$NameType;)Ljava/lang/String;
[compiled]
ffffffff6c4077e8 *
com/sun/org/apache/xerces/internal/impl/XMLDocumentScannerImpl$ContentDriver.scanRootElementHook()Z+4
ffffffff6cc42de0 *
*com/sun/org/apache/xerces/internal/impl/XMLDocumentFragmentScannerImpl$FragmentContentDriver.next()I
[compiled] +2224
ffffffff6c407b64 *
com/sun/org/apache/xerces/internal/impl/XMLDocumentScannerImpl$PrologDriver.next()I+196
ffffffff6c7bb9dc *
com/sun/org/apache/xerces/internal/impl/dtd/XMLDTDValidator.comment(Lcom/sun/org/apache/xerces/internal/xni/XMLString;Lcom/sun/org/apache/xerces/internal/xni/Augmentations;)V+103
ffffffff6c407b18 *
com/sun/org/apache/xerces/internal/impl/XMLDocumentFragmentScannerImpl.scanDocument(Z)Z+308
ffffffff6c407834 *
com/sun/org/apache/xerces/internal/parsers/XML11Configuration.parse(Z)Z+123
ffffffff6c4077e8 *
com/sun/org/apache/xerces/internal/parsers/XML11Configuration.parse(Lcom/sun/org/apache/xerces/internal/xni/parser/XMLInputSource;)V+29
ffffffff6c4080b4 *
com/sun/org/apache/xerces/internal/parsers/XMLParser.parse(Lcom/sun/org/apache/xerces/internal/xni/parser/XMLInputSource;)V+76
ffffffff6c408068 *
com/sun/org/apache/xerces/internal/parsers/AbstractSAXParser.parse(Lorg/xml/sax/InputSource;)V+43
ffffffff6c408068 *
com/sun/org/apache/xerces/internal/jaxp/SAXParserImpl$JAXPSAXParser.parse(Lorg/xml/sax/InputSource;)V+53
ffffffff6c4080b4 *
org/apache/tomcat/util/digester/Digester.parse(Ljava/io/InputStream;)Ljava/lang/Object;+18
ffffffff6c407f58 *
org/apache/tomcat/util/modeler/modules/MbeansDescriptorsDigesterSource.execute()V+61
ffffffff6c408068 *
org/apache/tomcat/util/modeler/modules/MbeansDescriptorsDigesterSource.loadDescriptors(Lorg/apache/tomcat/util/modeler/Registry;Ljava/lang/String;Ljava/lang/Object;)Ljava/util/List;+11
ffffffff6c407f58 *
org/apache/tomcat/util/modeler/Registry.load(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/String;)Ljava/util/List;+215
ffffffff6c407f58 *
org/apache/tomcat/util/modeler/Registry.loadDescriptors(Ljava/lang/String;Ljava/lang/ClassLoader;)V+139
ffffffff6c408068 *
org/apache/tomcat/util/modeler/Registry.findDescriptor(Ljava/lang/Class;Ljava/lang/String;)V+102
ffffffff6c408068 *
org/apache/tomcat/util/modeler/Registry.findManagedBean(Ljava/lang/Object;Ljava/lang/Class;Ljava/lang/String;)Lorg/apache/tomcat/util/modeler/ManagedBean;+58
ffffffff6c407f58 *
org/apache/tomcat/util/modeler/Registry.registerComponent(Ljava/lang/Object;Ljavax/management/ObjectName;Ljava/lang/String;)V+89
ffffffff6c408068 * org/apache/catalina/core/StandardWrapper.load()V+112
ffffffff6c4080b4 *
org/apache/catalina/core/StandardContext.loadOnStartup([Lorg/apache/catalina/Container;)Z+157
ffffffff6c4077e8 *
org/apache/catalina/core/StandardContext.startInternal()V+1280
ffffffff6c408068 * org/apache/catalina/util/LifecycleBase.start()V+210
ffffffff6c4080b4 *
org/apache/catalina/core/ContainerBase$StartChild.call()Ljava/lang/Void;+4
ffffffff6c407f58 *
org/apache/catalina/core/ContainerBase$StartChild.call()Ljava/lang/Object;+1
ffffffff6c407fa4 * java/util/concurrent/FutureTask.run()V+42
ffffffff6c4080b4 *
java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
ffffffff6c408068 *
java/util/concurrent/ThreadPoolExecutor$Worker.run()V+5
Unfortunately gdb does give me the code line in apr where the crash
happens, so currently I don't know which address is broken.
I will do the same tests on Linux and will try to check against 1.2.19
and other OpenSSL versions. But that will take some time. Just wanted
to give a heads up, that something looks fishy on my boxes.
Regards,
Rainer
