Hi Trey, do you see any other error messages in the logs of your Elasticsearch node(s)? Graylog won't start writing into the Elasticsearch cluster again until the health of index "graylog2_6" (which is probably the current deflector target) is GREEN again.
Cheers, Jochen On Wednesday, 13 January 2016 19:08:41 UTC+1, Trey Dockendorf wrote: > > Our data center recently had a catastrophic power loss. Once everything > was back up Graylog refuses to start [1] and the issue seems to be a > corrupt elasticsearch indexes [2]. I've attempted rerouting the indexes > but that has not worked. I fear my only option is to delete the corrupt > indexes, but I'm unsure what kind of impact that will have on Graylog. > > Any advice is greatly appreciated. > > Thanks, > - Trey > > [1]: > 2016-01-13T12:01:24.886-06:00 WARN [BlockingBatchedESOutput] Error while > waiting for healthy Elasticsearch cluster. Not flushing. > java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get > healthy within timeout > at > org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:174) > at > org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:179) > at > org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:112) > at > org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105) > at > org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-01-13T12:01:51.526-06:00 INFO [IndexRetentionThread] Elasticsearch > cluster not available, skipping index retention checks. > > [2]: > # curl -XGET http://localhost:9200/_cat/shards > graylog2_6 0 p STARTED 2700869 398.6mb 192.168.200.93 gl-es01-esgl2 > graylog2_6 3 p STARTED 2694087 397mb 192.168.200.93 gl-es01-esgl2 > graylog2_6 1 p UNASSIGNED > graylog2_6 2 p UNASSIGNED > graylog2_4 0 p STARTED 5040142 796.3mb 192.168.200.93 gl-es01-esgl2 > graylog2_4 3 p STARTED 5003263 789.1mb 192.168.200.93 gl-es01-esgl2 > graylog2_4 1 p STARTED 5001091 788.6mb 192.168.200.93 gl-es01-esgl2 > graylog2_4 2 p STARTED 4958409 783.6mb 192.168.200.93 gl-es01-esgl2 > graylog2_5 0 p STARTED 5019303 743.4mb 192.168.200.93 gl-es01-esgl2 > graylog2_5 3 p STARTED 5001404 739.3mb 192.168.200.93 gl-es01-esgl2 > graylog2_5 1 p STARTED 5000525 739.3mb 192.168.200.93 gl-es01-esgl2 > graylog2_5 2 p STARTED 4979658 737.7mb 192.168.200.93 gl-es01-esgl2 > graylog2_2 0 p STARTED 5023249 985.4mb 192.168.200.93 gl-es01-esgl2 > graylog2_2 3 p STARTED 4999096 979.1mb 192.168.200.93 gl-es01-esgl2 > graylog2_2 1 p STARTED 5001476 980.1mb 192.168.200.93 gl-es01-esgl2 > graylog2_2 2 p STARTED 4976833 973.5mb 192.168.200.93 gl-es01-esgl2 > graylog2_3 0 p STARTED 5000546 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_3 3 p STARTED 4998766 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_3 1 p STARTED 4999378 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_3 2 p STARTED 5001326 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_0 0 p STARTED 3686796 819.8mb 192.168.200.93 gl-es01-esgl2 > graylog2_0 3 p STARTED 3655173 812.8mb 192.168.200.93 gl-es01-esgl2 > graylog2_0 1 p STARTED 3655623 813mb 192.168.200.93 gl-es01-esgl2 > graylog2_0 2 p STARTED 3625428 805.7mb 192.168.200.93 gl-es01-esgl2 > graylog2_1 0 p STARTED 5053805 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_1 3 p STARTED 5000588 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_1 1 p STARTED 5001749 1gb 192.168.200.93 gl-es01-esgl2 > graylog2_1 2 p STARTED 4943861 1gb 192.168.200.93 gl-es01-esgl2 > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/3480ebe5-1818-4ae2-a1a7-908f9ad99afd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
