It seems that one of your elasticsearch nodes has crashed, and that it lost a shard. This is often caused by not having enough memory for elasticsearch, I would monitor the memory usage of elasticsearch here.
Unfortunately in this case there's not much you can do, but to delete the graylog2_0 index from the node that fails to start. But the log you pasted is a elasticsearch log file, not a graylog2 log file. You will find the graylog2_0 directory in the data directory of the elasticsearch node running on 10.121.70.195. Best, Kay On Sun, May 18, 2014 at 1:09 AM, Jerry George <[email protected]> wrote: > Hi, > > I have following services running > > * elasticsearch is running > graylog2-web-interface running as pid 11796 > mongodb start/running, process 11778 > > However, > graylog2-server not running > > Following is log from /var/log/elasticsearch/graylog2.log > > [2014-05-17 18:46:53,092][INFO ][node ] [Triathlon] > stopping ... > [2014-05-17 18:46:53,127][INFO ][node ] [Triathlon] > stopped > [2014-05-17 18:46:53,127][INFO ][node ] [Triathlon] > closing ... > [2014-05-17 18:46:53,138][INFO ][node ] [Triathlon] > closed > [2014-05-17 18:46:55,398][INFO ][node ] [Vesta] > version[0.90.10], pid[11768], build[0a5781f/2014-01-10T10:18:37Z] > [2014-05-17 18:46:55,398][INFO ][node ] [Vesta] > initializing ... > [2014-05-17 18:46:55,407][INFO ][plugins ] [Vesta] loaded > [], sites [] > [2014-05-17 18:46:57,650][INFO ][node ] [Vesta] > initialized > [2014-05-17 18:46:57,650][INFO ][node ] [Vesta] starting > ... > [2014-05-17 18:46:57,716][INFO ][transport ] [Vesta] > bound_address {inet[/0.0.0.0:9300]}, publish_address > {inet[/10.121.70.195:9300]} > [2014-05-17 18:47:00,799][INFO ][cluster.service ] [Vesta] > new_master [Vesta][5hyFjUWHQ-GcJqYRpodmNQ][inet[/10.121.70.195:9300]], > reason: zen-disco-join (elected_as_master) > [2014-05-17 18:47:00,836][INFO ][discovery ] [Vesta] > graylog2/5hyFjUWHQ-GcJqYRpodmNQ > [2014-05-17 18:47:00,879][INFO ][http ] [Vesta] > bound_address {inet[/0.0.0.0:9200]}, publish_address > {inet[/10.121.70.195:9200]} > [2014-05-17 18:47:00,880][INFO ][node ] [Vesta] started > [2014-05-17 18:47:01,834][INFO ][gateway ] [Vesta] > recovered [1] indices into cluster_state > [2014-05-17 18:47:03,519][INFO ][cluster.service ] [Vesta] added > {[graylog2-server][N2agvoFmTSOLIN-0th-XGg][inet[/10.121.70.195:9350]]{client=true, > data=false, master=false},}, reason: zen-disco-receive(join from > node[[graylog2-server][N2agvoFmTSOLIN-0th-XGg][inet[/10.121.70.195:9350]]{client=true, > data=false, master=false}]) > [2014-05-17 18:47:03,563][WARN ][indices.cluster ] [Vesta] > [graylog2_0][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [graylog2_0][0] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:238) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.IllegalArgumentException: No type mapped for [0] > at > org.elasticsearch.index.translog.Translog$Operation$Type.fromId(Translog.java:216) > at > org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:34) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:215) > ... 4 more > [2014-05-17 18:47:03,574][WARN ][cluster.action.shard ] [Vesta] > [graylog2_0][0] sending failed shard for [graylog2_0][0], > node[5hyFjUWHQ-GcJqYRpodmNQ], [P], s[INITIALIZING], indexUUID > [3_GYM0kNT_CQBIVOrtWiDA], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2_0][0] failed to recover > shard]; nested: IllegalArgumentException[No type mapped for [0]]; ]] > [2014-05-17 18:47:03,574][WARN ][cluster.action.shard ] [Vesta] > [graylog2_0][0] received shard failed for [graylog2_0][0], > node[5hyFjUWHQ-GcJqYRpodmNQ], [P], s[INITIALIZING], indexUUID > [3_GYM0kNT_CQBIVOrtWiDA], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2_0][0] failed to recover > shard]; nested: IllegalArgumentException[No type mapped for [0]]; ]] > [2014-05-17 18:47:05,839][INFO ][cluster.service ] [Vesta] removed > {[graylog2-server][N2agvoFmTSOLIN-0th-XGg][inet[/10.121.70.195:9350]]{client=true, > data=false, master=false},}, reason: > zen-disco-node_failed([graylog2-server][N2agvoFmTSOLIN-0th-XGg][inet[/10.121.70.195:9350]]{client=true, > data=false, master=false}), reason transport disconnected (with verified > connect) > [2014-05-17 18:47:07,445][WARN ][indices.cluster ] [Vesta] > [graylog2_0][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [graylog2_0][0] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:238) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.IllegalArgumentException: No type mapped for [0] > at > org.elasticsearch.index.translog.Translog$Operation$Type.fromId(Translog.java:216) > at > org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:34) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:215) > ... 4 more > [2014-05-17 18:47:07,451][WARN ][cluster.action.shard ] [Vesta] > [graylog2_0][0] sending failed shard for [graylog2_0][0], > node[5hyFjUWHQ-GcJqYRpodmNQ], [P], s[INITIALIZING], indexUUID > [3_GYM0kNT_CQBIVOrtWiDA], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2_0][0] failed to recover > shard]; nested: IllegalArgumentException[No type mapped for [0]]; ]] > [2014-05-17 18:47:07,451][WARN ][cluster.action.shard ] [Vesta] > [graylog2_0][0] received shard failed for [graylog2_0][0], > node[5hyFjUWHQ-GcJqYRpodmNQ], [P], s[INITIALIZING], indexUUID > [3_GYM0kNT_CQBIVOrtWiDA], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2_0][0] failed to recover > shard]; nested: IllegalArgumentException[No type mapped for [0]]; ]] > > > Any advise? I saw some wiki pages, but all dead links? > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
