We observed a situation recently where a large increase in the amount of data being ingested into ES caused log files in /var to fill to capacity on each node in a six node cluster.
Attempts to curl to a master VIP front-ended on the cluster would return a portion of the full json, but not all of it. Imagine expecting ten "pages" of some type of information and only receiving 2-3 when it suddenly stops. I first saw this when manually using curl on a Linux client. I mistakenly thought it might be a client issue or a load balancer or proxy issue between me and the the cluster. The next symptom came in the form of Elastic HQ being able to connect to the cluster but producing "*Error: Unable to Read Node List*". The third symptom came from Kibana being unable to pull up any dashboards, timing out and proclaiming that our ES cluster was down. Monitoring should have been enabled to monitor /var and we hadn't received any alerts on it, so of course we didn't think to stop and check any of the filesystems for out-of-space issues. After we noticed this and corrected the problem, all our problems went away. *The question is: Why would /var filling cause the cluster to return partial json in this manner?* Thanks, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/45780f2b-df8d-4780-a107-bd8c6a58d325%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
