Modifying SinkRunner to be pluggable to allow for failover/replication.
-----------------------------------------------------------------------
Key: FLUME-952
URL: https://issues.apache.org/jira/browse/FLUME-952
Project: Flume
Issue Type: Brainstorming
Components: Sinks+Sources
Reporter: Juhani Connolly
Fix For: v1.1.0
Implementing the failover sink runner the following was suggested:
1. This needs to be implemented on top of FLUME-949 which deals with removing
the notion of a PollableSink altogether. As a result, the SinkRunner will
become a concrete implementation that can then allow different sink handling
policies - such as either a failover policy (needed for this issue), or load
balancing policy (not needed for this issue). Hence the policy part needs to be
pluggable rather than the sink runner itself. An example of such a construct is
the ChannelSelector and ChannelProcessor implementations.
In Flume-865 I have implemented FailoverSinkRunner as a separate runner, but I
am open to the idea of making it pluggable if it makes the code more
maintainable.
As is, there are many differences between the requirements for Failover and a
normal Sink runner, including configuration, initialisation, shutdown, error
handling and event processing. If we were to make this pluggable, many hooks
would be needed and I don't think there is that much common behavior that
warrants using a pluggable system rather than just a solid base class.
- Adding a new sink to a runner, with configuration variables(such as priority
or weight)
- Policy for handling process: should this just return a list of sinks to
process like ChannelSelector and hand off the processing to Process? I think
that the specific failover policy for each type of runner will be different so
this feels awkward. I would personally prefer to just pass the process call to
the pluggable component and let it be responsible for calling process on the
correct sinks, as well as handling errors.
Right now I am not convinced for the need to make SinkRunner pluggable, but I
would be interested to hear other peoples opinions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
