I am not very familiar with ELB, but if you do have a patch that will make Flume work better - I’ll be glad to review it. I think the HDFS, HBase, Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but different impl).
Thanks, Hari On Tue, Mar 31, 2015 at 9:27 AM, Ashish <[email protected]> wrote: > Does ELB provide any kind of status code when it rejects the request > or it just doesn't send any response? > Perhaps adding a JIRA would be great. Doesn't ELB times out connection > which are Idle for more than 60 sec? > thanks > ashish > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic > <[email protected]> wrote: >> Hi, >> >> I don't have a very good bug report, but I want to share one problem with >> Flume we observed the other day. Specifically, with Elasticsearch Sink and >> AWS ELB, but others may have this problem, too. >> >> In short, ELB has this notion of a Spillover where it can reject requests >> when it can't handle the load. Unfortunately, it doesn't actually signal >> this by closing the connection, so Flume's connection remains open, I >> believe in some java.net....... readSocket(....) method. >> >> To work around this we had to add a socket timeout in our custom version of >> Flume's Elasticsearch Sink. >> >> I think it's worth fixing this in Flume and checking if other Sinks need >> this sort of timeout. >> >> HTH >> >> Otis >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http://sematext.com/ > -- > thanks > ashish > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal
