On 19/07/2022 23:16, Rainer Jung wrote:
Roughly the same pattern I saw for TC 10.0 now also seen for TC 10.1.
A failure rate of 1 in 50 is going to make any testing a relatively slow
process. We are going to need at least 250 test runs to get a reasonable
degree of certainty in the results.
I agree that it seems likely that the Connector shutdown code isn't 100%
robust. I took a quick look and cleanup seems to be triggered by GC
which should avoid these sorts of issues. This one may take a while to
track down.
Did you have any success in narrowing down the scope of the tests while
still reproducing the issue?
Mark
Am 18.07.2022 um 12:09 schrieb Rainer Jung:
Hi there,
this is just an info, at this point probably not a showstopper. The
topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
after TLS unit tests.
Details:
I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
I ran the test for a variety of OpenJDK builds (Adoptium, Zulu,
Oracle, RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17 and
current 19).
The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL 7
and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
I only ran about 150 test classes (for NIO and also for NIO2), because
I also ran the full unit tests (about 450 classes) for JSSE and didn't
want to rerun all tests for time and efficiency reasons.
For TC 10 I observed crashes in TLS tests during shutdown: Out of the
roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
observe a single one. Tests for TC 10.1 are ongoing, until now no
crash, but it is a bit early for a final result. I think the crashes
are not new. All hapened in the TLS tests in org.apache.tomcat.util.net.
The list of crashes I saw for TC 10.0.23:
RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
openjdk version "1.8.0_332-ea"
OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
double free or corruption (!prev): 0x00007f473c19df50
======= Backtrace: =========
/lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
[0x7f472d018427]
RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
corrupted double-linked list: 0x00007f6bb8001d10
======= Backtrace: =========
/lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
/lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
[0x7f6bd572249a]
SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
java version "1.8.0_331"
Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
double free or corruption (!prev): 0x00007fbf88c1de10
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
/lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
[0x7fbf770264a7]
SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
double free or corruption (!prev): 0x00007f4f6bb93040
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
/lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
[0x7f4f508b88b0]
RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
Since they are rare and happen in various tests and version
combinations, it seems the general shutdown behavior w.r.t. the
library is not yet perfect.
Once the tests for 10.1 complete, I will see, whether I can force the
crashes more often by focusing on the TLS tests in
org.apache.tomcat.util.net.
Best regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
