----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6939/#review11083 -----------------------------------------------------------
Awesome... we definitely need exponential backoff on the load balancing sink processor. Only thing is, I wonder if we shouldn't just add it to the two existing selectors instead of implement new ones that essentially do the same thing. How about modifying the constructors to accept a boolean (backoffOnFailure) instead? Other than that it looks good, I saw only a couple minor edge case issues noted below. flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java <https://reviews.apache.org/r/6939/#comment23813> I think we need to watch out here for when sequentialFails > 63 because it will go negative and then zero. Maybe add a Math.min() on sequentialFails itself with some reasonable value like 20. flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java <https://reviews.apache.org/r/6939/#comment23814> Same shift issue as above - Mike Percy On Sept. 6, 2012, 4:15 a.m., Juhani Connolly wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6939/ > ----------------------------------------------------------- > > (Updated Sept. 6, 2012, 4:15 a.m.) > > > Review request for Flume. > > > Description > ------- > > Added a callback to the SinkSelector interface to inform selectors of sink > failures, adding a noop stub to the abstract source so existing selectors > don't need changes. > > The new selector maintains counts of sequential failures as well as times for > intended recovery and last failure time, which it uses to decide whether or > not a sink should be added to the returned iterator. The iterator is > generated in such a way that the round robin remains balanced. > > > This addresses bug FLUME-1541. > https://issues.apache.org/jira/browse/FLUME-1541 > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java > 63397a5 > > flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java > 18d4509 > > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java > 1e9c94e > > Diff: https://reviews.apache.org/r/6939/diff/ > > > Testing > ------- > > Added 3 new tests to verify that dynamic rebalancing is done, that the sink > becomes available again after timeout, and that the timeout increases with > sequential failures > > All tests pass > > > Thanks, > > Juhani Connolly > >
