Our data center recently had a catastrophic power loss. Once everything was back up Graylog refuses to start [1] and the issue seems to be a corrupt elasticsearch indexes [2]. I've attempted rerouting the indexes but that has not worked. I fear my only option is to delete the corrupt indexes, but I'm unsure what kind of impact that will have on Graylog.
Any advice is greatly appreciated. Thanks, - Trey [1]: 2016-01-13T12:01:24.886-06:00 WARN [BlockingBatchedESOutput] Error while waiting for healthy Elasticsearch cluster. Not flushing. java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get healthy within timeout at org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:174) at org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:179) at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:112) at org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105) at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-01-13T12:01:51.526-06:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. [2]: # curl -XGET http://localhost:9200/_cat/shards graylog2_6 0 p STARTED 2700869 398.6mb 192.168.200.93 gl-es01-esgl2 graylog2_6 3 p STARTED 2694087 397mb 192.168.200.93 gl-es01-esgl2 graylog2_6 1 p UNASSIGNED graylog2_6 2 p UNASSIGNED graylog2_4 0 p STARTED 5040142 796.3mb 192.168.200.93 gl-es01-esgl2 graylog2_4 3 p STARTED 5003263 789.1mb 192.168.200.93 gl-es01-esgl2 graylog2_4 1 p STARTED 5001091 788.6mb 192.168.200.93 gl-es01-esgl2 graylog2_4 2 p STARTED 4958409 783.6mb 192.168.200.93 gl-es01-esgl2 graylog2_5 0 p STARTED 5019303 743.4mb 192.168.200.93 gl-es01-esgl2 graylog2_5 3 p STARTED 5001404 739.3mb 192.168.200.93 gl-es01-esgl2 graylog2_5 1 p STARTED 5000525 739.3mb 192.168.200.93 gl-es01-esgl2 graylog2_5 2 p STARTED 4979658 737.7mb 192.168.200.93 gl-es01-esgl2 graylog2_2 0 p STARTED 5023249 985.4mb 192.168.200.93 gl-es01-esgl2 graylog2_2 3 p STARTED 4999096 979.1mb 192.168.200.93 gl-es01-esgl2 graylog2_2 1 p STARTED 5001476 980.1mb 192.168.200.93 gl-es01-esgl2 graylog2_2 2 p STARTED 4976833 973.5mb 192.168.200.93 gl-es01-esgl2 graylog2_3 0 p STARTED 5000546 1gb 192.168.200.93 gl-es01-esgl2 graylog2_3 3 p STARTED 4998766 1gb 192.168.200.93 gl-es01-esgl2 graylog2_3 1 p STARTED 4999378 1gb 192.168.200.93 gl-es01-esgl2 graylog2_3 2 p STARTED 5001326 1gb 192.168.200.93 gl-es01-esgl2 graylog2_0 0 p STARTED 3686796 819.8mb 192.168.200.93 gl-es01-esgl2 graylog2_0 3 p STARTED 3655173 812.8mb 192.168.200.93 gl-es01-esgl2 graylog2_0 1 p STARTED 3655623 813mb 192.168.200.93 gl-es01-esgl2 graylog2_0 2 p STARTED 3625428 805.7mb 192.168.200.93 gl-es01-esgl2 graylog2_1 0 p STARTED 5053805 1gb 192.168.200.93 gl-es01-esgl2 graylog2_1 3 p STARTED 5000588 1gb 192.168.200.93 gl-es01-esgl2 graylog2_1 1 p STARTED 5001749 1gb 192.168.200.93 gl-es01-esgl2 graylog2_1 2 p STARTED 4943861 1gb 192.168.200.93 gl-es01-esgl2 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d72f0c6c-87cc-4edb-a167-2f5115501e0e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
