On my system, `cat /proc/sys/fs/file-nr` reports 24838 used FDs, and that goes up to 29030 when running the infinispan-compatibility-mode-it tests. Since there are only 78 tests, a leak is quite possible, but I wouldn't say 4K open files (or sockets, more likely) really is a deal-breaker.
Cheers Dan On Thu, Feb 23, 2017 at 1:31 AM, Sanne Grinovero <sa...@infinispan.org> wrote: > On 22 February 2017 at 21:20, Dennis Reed <der...@redhat.com> wrote: >> Are those actually 2 million *unique* descriptors? >> >> I've seen lsof output that listed many duplicates for the same file >> descriptor (one for each thread?), making the list appear much larger >> than it really was. > > Good point! You're right, I verified and all instances of e.g. the > jgroups jar were using the same FD, just a different thread id. > > This is the full error I'm having, when running the tests from the > "infinispan-compatibility-mode-it" maven module: > > > java.lang.IllegalStateException: failed to create a child event loop > at > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) > at > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) > at > io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:51) > at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:87) > at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:82) > at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:63) > at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:51) > at > org.jboss.resteasy.plugins.server.netty.NettyJaxrsServer.start(NettyJaxrsServer.java:239) > at org.infinispan.rest.NettyRestServer.start(NettyRestServer.java:81) > at > org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestCache(CompatibilityCacheFactory.java:199) > at > org.infinispan.it.compatibility.CompatibilityCacheFactory.createRestMemcachedCaches(CompatibilityCacheFactory.java:137) > at > org.infinispan.it.compatibility.CompatibilityCacheFactory.setup(CompatibilityCacheFactory.java:123) > at > org.infinispan.it.compatibility.ByteArrayKeyReplEmbeddedHotRodTest.setup(ByteArrayKeyReplEmbeddedHotRodTest.java:87) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84) > at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564) > at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213) > at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138) > at > org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175) > at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107) > at org.testng.TestRunner.privateRun(TestRunner.java:767) > at org.testng.TestRunner.run(TestRunner.java:617) > at org.testng.SuiteRunner.runTest(SuiteRunner.java:348) > at org.testng.SuiteRunner.access$000(SuiteRunner.java:38) > at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382) > at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: io.netty.channel.ChannelException: failed to open a new selector > at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:157) > at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:148) > at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:126) > at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) > at > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) > ... 32 more > Caused by: java.io.IOException: Too many open files > at sun.nio.ch.IOUtil.makePipe(Native Method) > at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) > at > sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) > at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:155) > ... 36 more > > Now that I know which metrics to look at, I see that before running > this specific module my system is consuming about 12K FDs, and > occasionally it just stays around that same level and the integration > tests will pass w/o any failure. > However most of the times when I run this module specifically, I see > the FD consumption increase during the test run and eventually fail > with the above error. The last sample I could take before failing was > around 15,5K, and not surprising as my limit is set to 16384.. so I > guess it could have attempted to grow further. > > Tomorrow I'll try with higher limits, to see if I'm running with > barely enough for this testsuite or if I see it growing more. Still > sounds like a leak though, as the increase is quite significant.. > > Thanks, > Sanne > >> >> -Dennis >> >> >> On 02/22/2017 02:25 PM, Sanne Grinovero wrote: >>> Hi all, >>> >>> our documentation suggest to raise the file limits to about 16K: >>> http://infinispan.org/docs/stable/contributing/contributing.html#running_and_writing_tests >>> >>> I already have this setup since years, yet I've been noticing errors such >>> as: >>> >>> "Caused by: java.io.IOException: Too many open files" >>> >>> Today I decided to finally have a look, and I see that while running >>> the testsuite, my system's consumption of file descriptor raises >>> continuously, up to more than 2 millions. >>> (When not running the suite, I'm consuming 200K - that's including >>> IDEs and other FD hungry systems like Chrome) >>> >>> Trying to get some samples of these file descriptors, it looks like >>> it's really about open files. Jar files to be more precise. >>> >>> What puzzles me is that taking just one jar - jgroups for example - I >>> can count 7852 open instances of it, but distributed among a handful >>> of processes only. >>> >>> My guess is classloaders aren't being closed? >>> >>> Also: why did nobody else notice problems? Do you all have >>> reconfigured your system for unlimited FDs? >>> >>> Thanks, >>> Sanne >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
