Hi all:
I get the same error:
2014-08-14 13:12:15,190 ERROR:
org.graylog2.periodical.ClusterHealthCheckThread - *Unable to find own
node: *
*org.graylog2.cluster.NodeNotFoundException: Did not find our own node.
This should never happen*.
at org.graylog2.cluster.Node.thisNode(Node.java:95)
at
org.graylog2.periodical.ClusterHealthCheckThread.getNotification(ClusterHealthCheckThread.java:34)
at
org.graylog2.periodical.ClusterHealthCheckThread.run(ClusterHealthCheckThread.java:23)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I know the superficial reason is the Full Garbage Collector, however, our
JVM is set to 230GB and we upgraded to latest version of graylog2
# jps -v
5441 graylog2-server.jar -Xms230G -Xmx230G -XX:NewRatio=20
-XX:+UseConcMarkSweepGC -XX:SurvivorRatio=20 -verbose:gc
# dpkg -l |grep graylog2
ii graylog2-server 0.20.6-1 all
Graylog2 Loggingframework - Server
I really doubt that the root cause is memory leak in java code.
Any help or comment is welcome!
Best regards
Guanqing, Lu
On Wednesday, July 2, 2014 8:49:36 PM UTC-4, Boden Garman wrote:
>
> Hi all,
>
> We've had an issue with our graylog2 installation where the server becomes
> unresponsive. There's a few messages in the logs, but this one is my
> favourite :)
>
> 2014-07-03 09:59:01,831 ERROR:
> org.graylog2.periodical.ClusterHealthCheckThread - Unable to find own node:
> org.graylog2.cluster.NodeNotFoundException: Did not find our own node.
> This should never happen.
>
> before that one we get
>
> 2014-07-03 09:45:39,244 ERROR:
> org.graylog2.outputs.BatchedElasticSearchOutput - Unable to flush message
> buffer: org.e
> lasticsearch.discovery.MasterNotDiscoveredException: waited for [30s] -
> [org.elasticsearch.action.support.master.Trans
> portMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180),
>
> org.elasticsearch.cluster.serv
> ice.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483),
>
> java.util.concurrent.ThreadPoolExecutor
> .runWorker(ThreadPoolExecutor.java:1145),
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15), java.lang.Thread.run(Thread.java:744)]
>
> and I think the root cause is this out of heap issue
>
> 2014-07-03 06:07:04,477 WARN :
> org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by
> a user hand
> ler while handling an exception event ([id: 0x336e4787, /127.0.0.1:58514
> => /127.0.0.1:12900] EXCEPTION: java.io.IOExc
> eption: Broken pipe)
> *java.lang.OutOfMemoryError: Java heap space*
> 2014-07-03 06:05:06,197 INFO :
> org.graylog2.jersey.container.netty.NettyContainer - Not writing any
> response, channel is already closed.
>
> We've increased the heap space the JVM can allocate to 1G but this only
> delays the inevitable.
>
> The server then uses 100% CPU on all cores and is unresponsive to stopping
> and can only be killed.
>
> Running the following versions on Debian Wheezy with openjdk7
> root@graylog:~# dpkg --list | grep graylog
> ii graylog2-server 0.20.3-1 all
> Graylog2 Loggingframework - Server
> ii graylog2-stream-dashboard 0.90.0-1 all
> Graylog2 Loggingframework - Stream Dashboard
> ii graylog2-web 0.20.3-1 all
> Graylog2 Loggingframework - Web Interface
>
> Any suggestions? Cheers!
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
