Hi, Unfortunately don't have a patch, though I think this is just 1 method that needs to be added. Don't think ELB provides the info needed, so one needs to just have timeout.
Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan <[email protected] > wrote: > I am not very familiar with ELB, but if you do have a patch that will make > Flume work better - I’ll be glad to review it. I think the HDFS, HBase, > Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but > different impl). > > > > > Thanks, Hari > > On Tue, Mar 31, 2015 at 9:27 AM, Ashish <[email protected]> wrote: > > > Does ELB provide any kind of status code when it rejects the request > > or it just doesn't send any response? > > Perhaps adding a JIRA would be great. Doesn't ELB times out connection > > which are Idle for more than 60 sec? > > thanks > > ashish > > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic > > <[email protected]> wrote: > >> Hi, > >> > >> I don't have a very good bug report, but I want to share one problem > with > >> Flume we observed the other day. Specifically, with Elasticsearch Sink > and > >> AWS ELB, but others may have this problem, too. > >> > >> In short, ELB has this notion of a Spillover where it can reject > requests > >> when it can't handle the load. Unfortunately, it doesn't actually > signal > >> this by closing the connection, so Flume's connection remains open, I > >> believe in some java.net....... readSocket(....) method. > >> > >> To work around this we had to add a socket timeout in our custom > version of > >> Flume's Elasticsearch Sink. > >> > >> I think it's worth fixing this in Flume and checking if other Sinks need > >> this sort of timeout. > >> > >> HTH > >> > >> Otis > >> -- > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > >> Solr & Elasticsearch Support * http://sematext.com/ > > -- > > thanks > > ashish > > Blog: http://www.ashishpaliwal.com/blog > > My Photo Galleries: http://www.pbase.com/ashishpaliwal >
