I've got a pretty good sized box collecting logs for a number of sources (about a dozen or so). Actually two instances were running on this box (one production and the other a testing environment)
After adding a dozen more log files i'm running into this error (87 logs for PROD and 70 or so for Testing) Nov 26, 2013 10:25:27 AM org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink WARNING: Failed to accept a connection. java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.start(AbstractNioWorker.java:160) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.register(AbstractNioWorker.java:131) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.registerAcceptedChannel(NioServerSocketPipelineSink.java:269) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:231) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I have been able to confirm that they are not bumping into the OS ulimit, and increasing the sizes of the JVM does not make this go away (JAVA_OPTS="-Xms1g -Xmx6g -Dcom.sun.management.jmxremot"). I ended up having to shut down the testing environment instance so as not to lose any more production logs. The box is 16 CPU with 16G RAM using local disks, running RHEL5.x Anytime process threads exceed ~930 (the results of ps -eLF -u flume | wc -l )the above error appears...if allowed to continue ultimately crashes the box. Any ideas on resolving this issue or have I run into a thread limit that cannot be gotten around without adding another server? Thanks, Dave
