Hi all,
We've had an issue with our graylog2 installation where the server becomes
unresponsive. There's a few messages in the logs, but this one is my
favourite :)
2014-07-03 09:59:01,831 ERROR:
org.graylog2.periodical.ClusterHealthCheckThread - Unable to find own node:
org.graylog2.cluster.NodeNotFoundException: Did not find our own node. This
should never happen.
before that one we get
2014-07-03 09:45:39,244 ERROR:
org.graylog2.outputs.BatchedElasticSearchOutput - Unable to flush message
buffer: org.e
lasticsearch.discovery.MasterNotDiscoveredException: waited for [30s] -
[org.elasticsearch.action.support.master.Trans
portMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180),
org.elasticsearch.cluster.serv
ice.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483),
java.util.concurrent.ThreadPoolExecutor
.runWorker(ThreadPoolExecutor.java:1145),
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15), java.lang.Thread.run(Thread.java:744)]
and I think the root cause is this out of heap issue
2014-07-03 06:07:04,477 WARN :
org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by
a user hand
ler while handling an exception event ([id: 0x336e4787, /127.0.0.1:58514 =>
/127.0.0.1:12900] EXCEPTION: java.io.IOExc
eption: Broken pipe)
*java.lang.OutOfMemoryError: Java heap space*
2014-07-03 06:05:06,197 INFO :
org.graylog2.jersey.container.netty.NettyContainer - Not writing any
response, channel is already closed.
We've increased the heap space the JVM can allocate to 1G but this only
delays the inevitable.
The server then uses 100% CPU on all cores and is unresponsive to stopping
and can only be killed.
Running the following versions on Debian Wheezy with openjdk7
root@graylog:~# dpkg --list | grep graylog
ii graylog2-server 0.20.3-1 all
Graylog2 Loggingframework - Server
ii graylog2-stream-dashboard 0.90.0-1 all
Graylog2 Loggingframework - Stream Dashboard
ii graylog2-web 0.20.3-1 all
Graylog2 Loggingframework - Web Interface
Any suggestions? Cheers!
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
