You should really ask this on the Logstash list - https://groups.google.com/forum/#!forum/logstash-users
Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 26 August 2014 00:49, Shih-Peng Lin <[email protected]> wrote: > I am using LogStash to collect the logs from my service. The volume of > the data is so large (20GB/day) that I am afraid that some of the data will > be dropped at peak time. > > So I asked question > <http://stackoverflow.com/questions/25397148/how-to-monitor-if-logstash-is-fully-loaded/25397505#25397505> > in > Stack Overflow and decided to add a Redis as a buffer between ELB and > LogStash to prevent data loss. > > However, I am curious about *when will LogStash exceed the queue capacity > and drop messages?* > > Because I've done some experiments and the result shows that LogStash can > completely process all the data without any loss, e.g., local file (a 20GB > text file) --> LogStash --> local file, netcat --> LogStash --> local file. > > Can someone give me a solid example (or scenario, if any) when LogStash > eventually drops messages? So I can have a better understanding about why > we need a buffer in front of it. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/57ad4bed-de0e-442a-bb40-a7d1079a148d%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aJdffkGA135wQERfMjYdRZdTdDMXa11NHXOfvLHOpy9w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
