On Sun, Jan 27, 2019 at 2:18 AM Rainer Jung <rainer.j...@kippdata.de> wrote:
> When running the unit tests for trunk using Java 11 and APR connector I > run out of memory, first in threads, later in heap. The prpoblematic > tests are TestDefaultServletEncodingWithBom and > TestDefaultServletEncodingWithoutBom. > > A thread dumps shows 540 sendfile threads like the following: > > [junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon prio=5 > There should be one sendfile thread (at most) per APR connector. 540 is only a static counter to avoid running into a connector name conflict when the auto port is used. The sendfile thread is normally started and stopped with the connector. There's probably another cause for this apparent resource leak (I don't get the error on my OpenJDK 11). The counter is a great indication of the lifecycle of the JVM. Rémy > os_prio=0 cpu=0.34ms elapsed=8031.17s tid=0x00007fd4b8171800 nid=0x1249 > in Object.wait() [0x00007fd497313000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on <no object reference available> > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116) > [junit] - waiting to re-lock in wait() <0x00000000f7b38278> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834) > [junit] > > Is it expected, that those tests use that many threads in parallel? > > The later heap memory OOME is probably also due to resource constraints > on that VM. The memory when the OOME happens is: > > [junit] Heap > [junit] def new generation total 76800K, used 67531K > [0x00000000f0600000, 0x00000000f5950000, 0x00000000f5950000) > [junit] eden space 68288K, 98% used [0x00000000f0600000, > 0x00000000f47f2e38, 0x00000000f48b0000) > [junit] from space 8512K, 0% used [0x00000000f48b0000, > 0x00000000f48b0000, 0x00000000f5100000) > [junit] to space 8512K, 0% used [0x00000000f5100000, > 0x00000000f5100000, 0x00000000f5950000) > [junit] tenured generation total 170688K, used 170687K > [0x00000000f5950000, 0x0000000100000000, 0x0000000100000000) > [junit] the space 170688K, 99% used [0x00000000f5950000, > 0x00000000ffffffd0, 0x0000000100000000, 0x0000000100000000) > [junit] Metaspace used 25248K, capacity 25384K, committed > 26624K, reserved 26624K > > In addition I see native errors (tcnative 1.2.21): > > [junit] Exception in thread "Catalina-utility-1" > java.lang.OutOfMemoryError: unable to create native thread: possibly out > of memory or process/resource limits reached > [junit] at java.base/java.lang.Thread.start0(Native Method) > [junit] [156.313s][warning][os,thread] Failed to start thread - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, > guardsize: 0k, detached. > [junit] [156.314s][warning][os,thread] Failed to start thread - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, > guardsize: 0k, detached. > [junit] at java.base/java.lang.Thread.start(Thread.java:803) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [junit] at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > [junit] at java.base/java.lang.Thread.run(Thread.java:834) > > an then later after many of those: > > [junit] 26-Jan-2019 23:34:12.475 SEVERE [main] > org.apache.catalina.core.ContainerBase.startInternal A child container > failed during start > [junit] java.util.concurrent.ExecutionException: > java.lang.OutOfMemoryError: Java heap space > [junit] at > java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) > [junit] at > java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) > [junit] at > > org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) > [junit] at > > org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > > org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > > org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) > [junit] at > > org.apache.catalina.startup.TomcatBaseTest$TomcatWithFastSessionIDs.start(TomcatBaseTest.java:845) > > I do not know, whether the pthread errors happen after to many threads > were created or are the reson for the many threads to get created. > > Regards, > > Rainer > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
