Chris, This sounds very suspiciously like a problem we had. We set up an experimental local ELK server (one node in the cluster) and fed it with logstash. I was manually cleaning up older data using the Elasticsearch Head plug-in, but over one weekend the cluster got into a funky state. The curl API said it was Yellow, but ES Head showed Green, and queries were hanging.
This was a VM that was dedicated to ES with 1TB disk space (only about 2% was ever used at any point in time), 4 CPUs, and 24GB RAM (though the Java JVM was not tuned to take advantage of all of this memory). Kibana was hosted as a site plug-in, but its usage was very light. Though I had been playing around with increasing the size limit of responses way past the default of 500, and I'm sure the ES server bore the brunt of that. I stopped and restarted ES and everything went back to normal. I installed Curator to clean up older indices automatically, and the problem has never returned. (I have also stopped telling Kibana to ask for up to 50000 response documents on a query!) I suspect you're getting some sort of OOM condition and that's when things start looking odd. Anyway, OOM is just a wild guess. I wouldn't have mentioned something so nebulous, but the symptoms you have are strikingly close to the ones we saw. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ccf8ab3d-c89c-42da-95ba-1b25198fc445%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
