Server is an m4.4xlarge AWS machine. Built from the Graylog-provided AMIs. I routinely see this:
2015-08-08_18:11:48.95572 WARN [AbstractNioSelector] Unexpected exception in the selector loop. 2015-08-08_18:11:48.95574 java.lang.OutOfMemoryError: Direct buffer memory 2015-08-08_18:11:48.95575 at java.nio.Bits.reserveMemory(Bits.java:658) 2015-08-08_18:11:48.95576 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) 2015-08-08_18:11:48.95576 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 2015-08-08_18:11:48.95577 at org.jboss.netty.channel.socket.nio.SocketReceiveBufferAllocator.newBuffer(SocketReceiveBufferAllocator.java:64) 2015-08-08_18:11:48.95577 at org.jboss.netty.channel.socket.nio.SocketReceiveBufferAllocator.get(SocketReceiveBufferAllocator.java:44) 2015-08-08_18:11:48.95577 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:62) 2015-08-08_18:11:48.95577 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) 2015-08-08_18:11:48.95578 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) 2015-08-08_18:11:48.95578 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) 2015-08-08_18:11:48.95578 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) 2015-08-08_18:11:48.95579 at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) 2015-08-08_18:11:48.95579 at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) 2015-08-08_18:11:48.95580 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) 2015-08-08_18:11:48.95580 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-08-08_18:11:48.95581 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-08-08_18:11:48.95581 at java.lang.Thread.run(Thread.java:745) 2015-08-08_18:11:49.04681 WARN [AbstractNioSelector] Unexpected exception in the selector loop. 2015-08-08_18:11:49.04683 java.lang.OutOfMemoryError: Direct buffer memory 2015-08-08_18:11:49.04684 at java.nio.Bits.reserveMemory(Bits.java:658) 2015-08-08_18:11:49.04684 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) 2015-08-08_18:11:49.04684 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 2015-08-08_18:11:49.04685 at org.jboss.netty.channel.socket.nio.SocketReceiveBufferAllocator.newBuffer(SocketReceiveBufferAllocator.java:64) 2015-08-08_18:11:49.04685 at org.jboss.netty.channel.socket.nio.SocketReceiveBufferAllocator.get(SocketReceiveBufferAllocator.java:44) 2015-08-08_18:11:49.04685 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:62) 2015-08-08_18:11:49.04686 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) 2015-08-08_18:11:49.04686 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) 2015-08-08_18:11:49.04686 at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) 2015-08-08_18:11:49.04687 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) 2015-08-08_18:11:49.04687 at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) 2015-08-08_18:11:49.04687 at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) 2015-08-08_18:11:49.04688 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) 2015-08-08_18:11:49.04689 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-08-08_18:11:49.04689 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-08-08_18:11:49.04689 at java.lang.Thread.run(Thread.java:745) Server of course stops processing all messages. I've tried increasing the heap size in /opt/graylog/sv/graylog-server/run to -Xmx4g just for fun. It holds out longer, but eventually fails. This server is probably unable to keep up with the incoming message rate. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/606cc32b-c473-4582-8ac9-edaf1ccceb5f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
