[
https://issues.apache.org/jira/browse/FLUME-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205304#comment-13205304
]
Jarek Jarcec Cecho commented on FLUME-952:
------------------------------------------
bq. This is workable but feels unwieldy, no need for the extra state imo, I
think it is more transparent to give selector developers a function in the
interface they need to implement.
I tent to agree that having extra function for marking dead sink would make
much more sense.
bq. I don't think we can just keep hammering them every failed message.
I definitely agree here. Trying dead sinks with every message would kill the
performance. I was more thinking about configuration for selector to try "dead
sink" once a while (once a 15 minutes for example).
bq. What I would have liked to do is keep a list of dead sinks, and start up
another thread that could periodically poll them for recovery.
Seems as a good idea to me. However how do you want to resolve dead sinks? Are
you going to simply try process() method and see what happens?
bq. One thing I am sure of is that right now sink implementations are
inconsistent with one another.
I would suggest to create separate JIRA for that and fix discrepancy in
behavior prior creating selectors in SinkRunner.
bq. One possibility is that failed sinks should change their lifecycle state to
stopped and that all sinks would be required to make some kind of liveliness
check when starting.
This again seems as a good idea how to deal with dead sinks. Your suggested
thread that will keep an eye on dead sinks might then just simply call method
start() to see if the sink will be able to start.
> Modifying SinkRunner to be pluggable to allow for failover/replication.
> -----------------------------------------------------------------------
>
> Key: FLUME-952
> URL: https://issues.apache.org/jira/browse/FLUME-952
> Project: Flume
> Issue Type: Brainstorming
> Components: Sinks+Sources
> Reporter: Juhani Connolly
> Fix For: v1.1.0
>
>
> Implementing the failover sink runner the following was suggested:
> 1. This needs to be implemented on top of FLUME-949 which deals with removing
> the notion of a PollableSink altogether. As a result, the SinkRunner will
> become a concrete implementation that can then allow different sink handling
> policies - such as either a failover policy (needed for this issue), or load
> balancing policy (not needed for this issue). Hence the policy part needs to
> be pluggable rather than the sink runner itself. An example of such a
> construct is the ChannelSelector and ChannelProcessor implementations.
> In Flume-865 I have implemented FailoverSinkRunner as a separate runner, but
> I am open to the idea of making it pluggable if it makes the code more
> maintainable.
> As is, there are many differences between the requirements for Failover and a
> normal Sink runner, including configuration, initialisation, shutdown, error
> handling and event processing. If we were to make this pluggable, many hooks
> would be needed and I don't think there is that much common behavior that
> warrants using a pluggable system rather than just a solid base class.
> - Adding a new sink to a runner, with configuration variables(such as
> priority or weight)
> - Policy for handling process: should this just return a list of sinks to
> process like ChannelSelector and hand off the processing to Process? I think
> that the specific failover policy for each type of runner will be different
> so this feels awkward. I would personally prefer to just pass the process
> call to the pluggable component and let it be responsible for calling process
> on the correct sinks, as well as handling errors.
> Right now I am not convinced for the need to make SinkRunner pluggable, but I
> would be interested to hear other peoples opinions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
