[
https://issues.apache.org/jira/browse/FLUME-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205275#comment-13205275
]
Juhani Connolly commented on FLUME-952:
---------------------------------------
Hi Jarcec, thanks for your comments.
Overall it sounds reasonable, one of my sticking points is with communicating
sink failure as well as recovering failed sinks:
- I like the idea of changing it to multiple sinks and enforcing a limit for
specific selectors
- This configuration will invalidate existing configurations... Not that big a
deal I guess, I had just tried to maintain it with my previous idea(and this
one should result in a shorter config file)
- We need a mechanism to let the selector know a sink has failed. From there
the selector can mark it as dead, rebalancing if necessary. Other than that the
interface looks good to me
- My biggest concern is how we will test for sink recovery. Most sinks the only
way to know if they are working is to try to attempt to send data through them
> Modifying SinkRunner to be pluggable to allow for failover/replication.
> -----------------------------------------------------------------------
>
> Key: FLUME-952
> URL: https://issues.apache.org/jira/browse/FLUME-952
> Project: Flume
> Issue Type: Brainstorming
> Components: Sinks+Sources
> Reporter: Juhani Connolly
> Fix For: v1.1.0
>
>
> Implementing the failover sink runner the following was suggested:
> 1. This needs to be implemented on top of FLUME-949 which deals with removing
> the notion of a PollableSink altogether. As a result, the SinkRunner will
> become a concrete implementation that can then allow different sink handling
> policies - such as either a failover policy (needed for this issue), or load
> balancing policy (not needed for this issue). Hence the policy part needs to
> be pluggable rather than the sink runner itself. An example of such a
> construct is the ChannelSelector and ChannelProcessor implementations.
> In Flume-865 I have implemented FailoverSinkRunner as a separate runner, but
> I am open to the idea of making it pluggable if it makes the code more
> maintainable.
> As is, there are many differences between the requirements for Failover and a
> normal Sink runner, including configuration, initialisation, shutdown, error
> handling and event processing. If we were to make this pluggable, many hooks
> would be needed and I don't think there is that much common behavior that
> warrants using a pluggable system rather than just a solid base class.
> - Adding a new sink to a runner, with configuration variables(such as
> priority or weight)
> - Policy for handling process: should this just return a list of sinks to
> process like ChannelSelector and hand off the processing to Process? I think
> that the specific failover policy for each type of runner will be different
> so this feels awkward. I would personally prefer to just pass the process
> call to the pluggable component and let it be responsible for calling process
> on the correct sinks, as well as handling errors.
> Right now I am not convinced for the need to make SinkRunner pluggable, but I
> would be interested to hear other peoples opinions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
