Hi, when I got this kind of error the only thing that has solved the problem was to stop all graylog2-servers, restart mongodb, restart elasticsearch-Cluster, wait for elastic-search-Cluster to become "green" - and after that start graylog2-Cluster (and perhaps radio-Nodes...)
regards, Cornelius On Thursday, August 14, 2014 3:35:47 PM UTC+2, Guanqing Lu wrote: > > Hi all: > > I get the same error: > > 2014-08-14 13:12:15,190 ERROR: > org.graylog2.periodical.ClusterHealthCheckThread - *Unable to find own > node: * > > *org.graylog2.cluster.NodeNotFoundException: Did not find our own node. > This should never happen*. > > at org.graylog2.cluster.Node.thisNode(Node.java:95) > > at > org.graylog2.periodical.ClusterHealthCheckThread.getNotification(ClusterHealthCheckThread.java:34) > > > > at > org.graylog2.periodical.ClusterHealthCheckThread.run(ClusterHealthCheckThread.java:23) > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > at java.lang.Thread.run(Thread.java:745) > > > I know the superficial reason is the Full Garbage Collector, however, our > JVM is set to 230GB and we upgraded to latest version of graylog2 > > > # jps -v > > 5441 graylog2-server.jar -Xms230G -Xmx230G -XX:NewRatio=20 > -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=20 -verbose:gc > > > # dpkg -l |grep graylog2 > > ii graylog2-server 0.20.6-1 all > Graylog2 Loggingframework - Server > > > I really doubt that the root cause is memory leak in java code. > > > Any help or comment is welcome! > > > Best regards > > > Guanqing, Lu > > > > On Wednesday, July 2, 2014 8:49:36 PM UTC-4, Boden Garman wrote: >> >> Hi all, >> >> We've had an issue with our graylog2 installation where the server >> becomes unresponsive. There's a few messages in the logs, but this one is >> my favourite :) >> >> 2014-07-03 09:59:01,831 ERROR: >> org.graylog2.periodical.ClusterHealthCheckThread - Unable to find own node: >> org.graylog2.cluster.NodeNotFoundException: Did not find our own node. >> This should never happen. >> >> before that one we get >> >> 2014-07-03 09:45:39,244 ERROR: >> org.graylog2.outputs.BatchedElasticSearchOutput - Unable to flush message >> buffer: org.e >> lasticsearch.discovery.MasterNotDiscoveredException: waited for [30s] - >> [org.elasticsearch.action.support.master.Trans >> portMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180), >> >> org.elasticsearch.cluster.serv >> ice.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483), >> >> java.util.concurrent.ThreadPoolExecutor >> .runWorker(ThreadPoolExecutor.java:1145), >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 >> 15), java.lang.Thread.run(Thread.java:744)] >> >> and I think the root cause is this out of heap issue >> >> 2014-07-03 06:07:04,477 WARN : >> org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by >> a user hand >> ler while handling an exception event ([id: 0x336e4787, /127.0.0.1:58514 >> => /127.0.0.1:12900] EXCEPTION: java.io.IOExc >> eption: Broken pipe) >> *java.lang.OutOfMemoryError: Java heap space* >> 2014-07-03 06:05:06,197 INFO : >> org.graylog2.jersey.container.netty.NettyContainer - Not writing any >> response, channel is already closed. >> >> We've increased the heap space the JVM can allocate to 1G but this only >> delays the inevitable. >> >> The server then uses 100% CPU on all cores and is unresponsive to >> stopping and can only be killed. >> >> Running the following versions on Debian Wheezy with openjdk7 >> root@graylog:~# dpkg --list | grep graylog >> ii graylog2-server 0.20.3-1 all >> Graylog2 Loggingframework - Server >> ii graylog2-stream-dashboard 0.90.0-1 all >> Graylog2 Loggingframework - Stream Dashboard >> ii graylog2-web 0.20.3-1 all >> Graylog2 Loggingframework - Web Interface >> >> Any suggestions? Cheers! >> > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
