I checked on RHEL 7 where I have less resource constraints. Both unit
tests (TestDefaultServletEncodingWithBom and
TestDefaultServletEncodingWithoutBom) each create about 1300 of those
Sendfile threads during APR test and the threads are only
stopped/destroyed at the end of the test. That's probably a bit excessive.
I don't understand the lifecycle of the embedded TC during the unit
tests and the sendfile threads in APR well enough to be able to suggest
a fix.
Regards,
Rainer
Am 27.01.2019 um 15:10 schrieb Rainer Jung:
I see the same problem on RHEL 6, but only the thread part of it. It
does not escalate to a heap OOME, so the test "just" fails but does not
hang. I do not have a thread dump there, but on RHEL 6 it also only
happens sng the APR connector, so could well be the same APR Sendfile
thread problem.
Regards,
Rainer
Am 27.01.2019 um 02:18 schrieb Rainer Jung:
When running the unit tests for trunk using Java 11 and APR connector
I run out of memory, first in threads, later in heap. The prpoblematic
tests are TestDefaultServletEncodingWithBom and
TestDefaultServletEncodingWithoutBom.
A thread dumps shows 540 sendfile threads like the following:
[junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon
prio=5 os_prio=0 cpu=0.34ms elapsed=8031.17s tid=0x00007fd4b8171800
nid=0x1249 in Object.wait() [0x00007fd497313000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <no object reference available>
[junit] at
java.lang.Object.wait(java.base@11.0.2/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000f7b38278>
(a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at
java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
Is it expected, that those tests use that many threads in parallel?
The later heap memory OOME is probably also due to resource
constraints on that VM. The memory when the OOME happens is:
[junit] Heap
[junit] def new generation total 76800K, used 67531K
[0x00000000f0600000, 0x00000000f5950000, 0x00000000f5950000)
[junit] eden space 68288K, 98% used [0x00000000f0600000,
0x00000000f47f2e38, 0x00000000f48b0000)
[junit] from space 8512K, 0% used [0x00000000f48b0000,
0x00000000f48b0000, 0x00000000f5100000)
[junit] to space 8512K, 0% used [0x00000000f5100000,
0x00000000f5100000, 0x00000000f5950000)
[junit] tenured generation total 170688K, used 170687K
[0x00000000f5950000, 0x0000000100000000, 0x0000000100000000)
[junit] the space 170688K, 99% used [0x00000000f5950000,
0x00000000ffffffd0, 0x0000000100000000, 0x0000000100000000)
[junit] Metaspace used 25248K, capacity 25384K, committed
26624K, reserved 26624K
In addition I see native errors (tcnative 1.2.21):
[junit] Exception in thread "Catalina-utility-1"
java.lang.OutOfMemoryError: unable to create native thread: possibly
out of memory or process/resource limits reached
[junit] at java.base/java.lang.Thread.start0(Native Method)
[junit] [156.313s][warning][os,thread] Failed to start thread -
pthread_create failed (EAGAIN) for attributes: stacksize: 1024k,
guardsize: 0k, detached.
[junit] [156.314s][warning][os,thread] Failed to start thread -
pthread_create failed (EAGAIN) for attributes: stacksize: 1024k,
guardsize: 0k, detached.
[junit] at java.base/java.lang.Thread.start(Thread.java:803)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit] at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
[junit] at java.base/java.lang.Thread.run(Thread.java:834)
an then later after many of those:
[junit] 26-Jan-2019 23:34:12.475 SEVERE [main]
org.apache.catalina.core.ContainerBase.startInternal A child container
failed during start
[junit] java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
[junit] at
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at
java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
[junit] at
org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
[junit] at
org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.core.StandardService.startInternal(StandardService.java:423)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.startup.Tomcat.start(Tomcat.java:455)
[junit] at
org.apache.catalina.startup.TomcatBaseTest$TomcatWithFastSessionIDs.start(TomcatBaseTest.java:845)
I do not know, whether the pthread errors happen after to many threads
were created or are the reson for the many threads to get created.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
