I've just realised that this happens every 2 or so hours on the dot.
On Wednesday, 7 October 2015 11:22:31 UTC+1, Michel Laporte wrote: > > Hi Jochen, > > Is there any way to reduce the pauses from the Java GC? Or any way around > that? Someone from ES also told me it might be the GC pauses. > > Thanks > > On Monday, 5 October 2015 22:01:30 UTC+1, Jochen Schalanda wrote: >> >> Hi Michel, >> >> this looks like the Graylog server node is losing connection to the >> Elasticsearch cluster occasionally. It might be connected to long GC pauses >> in the Graylog server process. Do you see any error messages in the logs of >> your Graylog server node? >> >> You can also activate GC logging as described at >> https://plumbr.eu/blog/garbage-collection/understanding-garbage-collection-logs >> . >> >> >> Cheers, >> Jochen >> >> On Monday, 5 October 2015 18:24:26 UTC+2, Michel Laporte wrote: >>> >>> Hi, >>> >>> 2 Main Errors i keep noticing is: >>> >>> 2015-10-05 11:36:44,406][WARN ][discovery.zen.publish ] >>> [ess-lon-gray-003_master] timed out waiting for all nodes to process >>> published state [8625] (timeout [30s], pending nodes: >>> [[graylog2-server][IuYwswm-QGCfMoIIYEGyMA][ess-lon-gray-002][inet[/192.168.32.70:9350]]{client=true, >>> >>> data=false, master=false}, >>> [ess-lon-gray-002_slave][Gy1EMkaTSgilwenuqw5o7Q][ess-lon-gray-002][inet[/192.168.32.70:9300]]]) >>> >>> >>> & >>> >>> [2015-10-05 11:47:54,495][INFO ][cluster.service ] >>> [ess-lon-gray-003_master] removed >>> {[graylog2-server][IuYwswm-QGCfMoIIYEGyMA][ess-lon-gray-002][inet[/192.168.32.70:9350]]{client=true, >>> >>> data=false, master=false},}, reason: >>> zen-disco-node_failed([graylog2-server][IuYwswm-QGCfMoIIYEGyMA][ess-lon-gray-002][inet[/192.168.32.70:9350]]{client=true, >>> >>> data=false, master=false}), reason transport disconnected >>> >>> The second one states Transport Disconnected? >>> >>> >>> You reckon that would be an issue with ES 1.4.2? Im withholding >>> upgrading yet just incase it messes up my config. >>> >>> Thanks >>> >>> >>> On Monday, 5 October 2015 16:47:49 UTC+1, Jochen Schalanda wrote: >>>> >>>> Hi Michel, >>>> >>>> first of all I'd recommend upgrading to the latest stable version of >>>> Elasticsearch (which is Elasticsearch 1.7.2 >>>> <https://www.elastic.co/downloads/past-releases/elasticsearch-1-7-2> >>>> at the time of writing). >>>> >>>> Are there any error messages in the logs of your Elasticsearch nodes? >>>> >>>> >>>> Cheers, >>>> Jochen >>>> >>>> On Monday, 5 October 2015 17:16:07 UTC+2, Michel Laporte wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have 2 Graylog Servers both running ES instances. However, after 1-2 >>>>> days of running, my Elasticsearch cluster always goes "No Active Master" >>>>> (Using KOPF) for about 10-15 minutes, then comes back up. >>>>> >>>>> >>>>> When it comes back up, i can NEVER search for Sources. I get this >>>>> Error: >>>>> >>>>> >>>>> Could not load histogram data >>>>> Loading of histogram data failed with status: Bad Request >>>>> >>>>> >>>>> However, if i select time to " All Week, or All month", data shows up, >>>>> but sometimes doesn't show the messages when you query it from the >>>>> source. >>>>> The only way to fix this is to restart ES on the Master then everything >>>>> comes back up and works fine. >>>>> >>>>> >>>>> Could someone help me? >>>>> >>>>> So far i've increased Ram to 16GB on each Machine and dedicated 7GB >>>>> Heap to each one as i thought it would be RAM. However, after a few days >>>>> each machine has about 500-900MB of RAM free. They are both on Linux >>>>> Ubuntu >>>>> 14.04 Machines. >>>>> >>>>> >>>>> Thanks (Errors Below) >>>>> >>>>> >>>>> ES KOPF LOG: >>>>> [2015-10-05 13:44:48] >>>>> {"status":503,"name":"ess-lon-gray-002_slave","cluster_name":"graylogemea","version":{"number":"1.4.5","build_hash":"2aaf797f2a571dcb779a3b61180afe8390ab61f9","build_timestamp":"2015-04-27T08:06:06Z","build_snapshot":false,"lucene_version":"4.10.4"},"tagline":"You >>>>> >>>>> Know, for Search"} >>>>> [2015-10-05 13:44:48] No active master, switching to basic mode >>>>> >>>>> ------------------------------------- >>>>> essencedigital.com <http://www.essencedigital.com/> >>>>> Google+ <https://plus.google.com/102138558390623994587/about> • >>>>> Facebook <http://www.facebook.com/essencedigital> • Twitter >>>>> <https://twitter.com/essencedigital> • YouTube >>>>> <http://www.youtube.com/essencedigitalvideos> >>>>> >>>> -- ------------------------------------- essencedigital.com <http://www.essencedigital.com/> Google+ <https://plus.google.com/102138558390623994587/about> • Facebook <http://www.facebook.com/essencedigital> • Twitter <https://twitter.com/essencedigital> • YouTube <http://www.youtube.com/essencedigitalvideos> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/0868f948-e1a1-463f-a3b5-ce64db355630%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
