Hello, I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every few days, Kibana will stop showing logs at some random time in the middle of the night. Last night, the last log entry I received in Kibana was around 18:30. When I checked on the ES servers, it showed the master running and the secondary not running (from /sbin/service elasticsearch status), but I was able to do a curl on the localhost and it returned information. So not sure what's up with that. Anyway, when I do a status on the master node, I get this:
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' { "cluster_name" : "gis-elasticsearch", "status" : "red", "timed_out" : false, "number_of_nodes" : 6, "number_of_data_nodes" : 2, "active_primary_shards" : 186, "active_shards" : 194, "relocating_shards" : 0, "initializing_shards" : 7, "unassigned_shards" : 249 } When I view the indexes, via "ls ...nodes/0/indeces/" it shows all indexes being modified today for some reason and there are new file for today's date.So I think I'm starting to catch back up after I restarted both servers but not sure why it failed in the first place. When I look at the logs on the master, I only see 4 warning errors at 18:57 and then the 2ndary leaving the cluster. I don't see any logs on the secondary (Pistol) on why it stopped working or what truly happened. [2014-03-06 18:57:04,121][WARN ][transport ] [ElasticSearch Server1] Transport response handler not found of id [64147630] [2014-03-06 18:57:04,124][WARN ][transport ] [ElasticSearch Server1] Transport response handler not found of id [64147717] [2014-03-06 18:57:04,124][WARN ][transport ] [ElasticSearch Server1] Transport response handler not found of id [64147718] [2014-03-06 18:57:04,124][WARN ][transport ] [ElasticSearch Server1] Transport response handler not found of id [64147721] [2014-03-06 19:56:08,467][INFO ][cluster.service ] [ElasticSearch Server1] removed {[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.1.1.10:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.13.3.46:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout [2014-03-06 19:56:12,304][INFO ][cluster.service ] [ElasticSearch Server1] added {[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.1.1.10:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.13.3.46:9301]]{client=true, data=false}]) Any idea on additional logging or troubleshooting I can turn on to keep this from happening in the future? Since the shards are not caught up, right now I"m just seeing a lot o debug messages about failed to parse. I'm assuming that will be corrected once we catch up. [2014-03-07 10:06:52,235][DEBUG][action.search.type ] [ElasticSearch Server1] All shards failed for phase: [query] [2014-03-07 10:06:52,223][DEBUG][action.search.type ] [ElasticSearch Server1] [windows-2014.03.07][3], node[W6aEFbimR5G712ddG_G5yQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@74ecbbc6] lastShard [true] org.elasticsearch.search.SearchParseException: [windows-2014.03.07][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date_histogram":{"field":"@timestamp","interval":"10m"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"(ASA AND Deny)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1394118412373,"to":"now"}}}]}}}}}}}},"size":0}]] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a693898-16e3-449f-9bf5-6adc97251e09%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
