Hi Experts, >From past few days there is a problem in our graylog2. *Need your help to resolve the issue*
*Problem: while we are able to see the all messages, the recent is not working.* - On the home page we see this message o Error! Syntax error in search query or empty index. - On the health page, we see this message http://host:port/health Indices & Shards Information is not up to date! Seems like you have no Graylog2 node running. Informaton last updated at 02.04.2014 - 15:03:52 by hostname-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba - On the gralog2_recent, we see this message *There is no shards allocated* curl -X GET ' http://localhost:9200/graylog2_recent/message/_search?from=0&size=10&pretty' { "error" : "SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[_na_][graylog2_recent][0]: No active shards}{[_na_][graylog2_recent][1]: No active shards}{[_na_][graylog2_recent][2]: No active shards}{[_na_][graylog2_recent][3]: No active shards}]", "status" : 500 - While other indices are running just fine . all below provides me the result. curl -X GET ' http://localhost:9200/graylog2_1/message/_search?from=0&size=10&pretty' curl -X GET ' http://localhost:9200/graylog2_0/message/_search?from=0&size=10&pretty' curl -X GET ' http://localhost:9200/twitter/message/_search?from=0&size=10&pretty' - Here is the cluster health output [elastic@dm100opera001 elasticsearch]$ curl -XGET ' http://localhost:9200/_cluster/health?pretty=true' { "cluster_name" : "graylog2", "status" : "red", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 1, "active_primary_shards" : 11, "active_shards" : 11, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 11 }[elastic@dm100opera001 elasticsearch]$ - Here is the shards level status : *graylog2_1* curl -XGET ' http://localhost:9200/_cluster/health/graylog2_1?level=shards&pretty=true' { "cluster_name" : "graylog2", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 1, "active_primary_shards" : 4, "active_shards" : 4, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "indices" : { "graylog2_1" : { "status" : "green", "number_of_shards" : 4, "number_of_replicas" : 0, "active_primary_shards" : 4, "active_shards" : 4, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "shards" : { "0" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 }, "1" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 }, "2" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 }, "3" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } } } } - Here is the shards level status : *graylog2_0* curl -XGET ' http://localhost:9200/_cluster/health/graylog2_0?level=shards&pretty=true' { "cluster_name" : "graylog2", "status" : "red", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 1, "active_primary_shards" : 2, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "indices" : { "graylog2_0" : { * "status" : "red",* "number_of_shards" : 4, "number_of_replicas" : 0, "active_primary_shards" : 2, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "shards" : { "0" : { * "status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "1" : { * "status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "2" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 }, "3" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } } } } - Here is the shards level status : *graylog2_recent* curl -XGET ' http://localhost:9200/_cluster/health/graylog2_recent?level=shards&pretty=true ' { "cluster_name" : "graylog2", "*status" : "red",* "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 4, "indices" : { "graylog2_recent" : { * "status" : "red",* "number_of_shards" : 4, "number_of_replicas" : 0, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 4, "shards" : { "0" : { * "status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "1" : { "*status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "2" : { "*status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "3" : { * "status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } } } - Here is what we get from the graylog2 logs : Nodes: [graylog2-server][Q8PCsFLtTyCAdjCyVKKeEg][inet[/131.177.54.227:9350]]{client=true, data=false, master=false}, local [graylog2-node-<hostname>][w6NEUpNwRxugT6gFiu9IVw][inet[/131.177.54.227:9300]], master routing_table: -- index [graylog2_recent] ----shard_id [graylog2_recent][0] --------[graylog2_recent][0], node[null], [P], s[UNASSIGNED] ----shard_id [graylog2_recent][1] --------[graylog2_recent][1], node[null], [P], s[UNASSIGNED] ----shard_id [graylog2_recent][2] --------[graylog2_recent][2], node[null], [P], s[UNASSIGNED] ----shard_id [graylog2_recent][3] --------[graylog2_recent][3], node[null], [P], s[UNASSIGNED] -- index [twitter] ----shard_id [twitter][0] --------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][0], node[null], [R], s[UNASSIGNED] ----shard_id [twitter][1] --------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][1], node[null], [R], s[UNASSIGNED] ----shard_id [twitter][2] --------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][2], node[null], [R], s[UNASSIGNED] ----shard_id [twitter][3] --------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][3], node[null], [R], s[UNASSIGNED] ----shard_id [twitter][4] --------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][4], node[null], [R], s[UNASSIGNED] -- index [graylog2_0] ----shard_id [graylog2_0][0] --------[graylog2_0][0], node[null], [P], s[UNASSIGNED] ----shard_id [graylog2_0][1] --------[graylog2_0][1], node[null], [P], s[UNASSIGNED] ----shard_id [graylog2_0][2] --------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] ----shard_id [graylog2_0][3] --------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] -- index [graylog2_1] ----shard_id [graylog2_1][0] --------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] ----shard_id [graylog2_1][1] --------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] ----shard_id [graylog2_1][2] --------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] ----shard_id [graylog2_1][3] --------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] routing_nodes: -----node_id[w6NEUpNwRxugT6gFiu9IVw][V] --------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] --------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED] ---- unassigned --------[graylog2_recent][0], node[null], [P], s[UNASSIGNED] --------[graylog2_recent][1], node[null], [P], s[UNASSIGNED] --------[graylog2_recent][2], node[null], [P], s[UNASSIGNED] --------[graylog2_recent][3], node[null], [P], s[UNASSIGNED] --------[twitter][0], node[null], [R], s[UNASSIGNED] --------[twitter][1], node[null], [R], s[UNASSIGNED] --------[twitter][2], node[null], [R], s[UNASSIGNED] --------[twitter][3], node[null], [R], s[UNASSIGNED] --------[twitter][4], node[null], [R], s[UNASSIGNED] --------[graylog2_0][0], node[null], [P], s[UNASSIGNED] --------[graylog2_0][1], node[null], [P], s[UNASSIGNED] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
