I am using LogStash to collect the logs from my service. The volume of the data is so large (20GB/day) that I am afraid that some of the data will be dropped at peak time.
So I asked question <http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505> in Stack Overflow and decided to add a Redis as a buffer between ELB and LogStash to prevent data loss. However, I am curious about *when will LogStash exceed the queue capacity and drop messages?* Because I've done some experiments and the result shows that LogStash can completely process all the data without any loss, e.g., local file (a 20GB text file) --> LogStash --> local file, netcat --> LogStash --> local file. Can someone give me a solid example (or scenario, if any) when LogStash eventually drops messages? So I can have a better understanding about why we need a buffer in front of it. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
