[
https://issues.apache.org/jira/browse/FLUME-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205291#comment-13205291
]
Jarek Jarcec Cecho commented on FLUME-952:
------------------------------------------
Hi Juhani,
thank you very much for your feedback. Let me clarify some thoughts that I
wasn't able to describe enough yesterday night.
bq. - I like the idea of changing it to multiple sinks and enforcing a limit
for specific selectors
Thanks :)
bq. - This configuration will invalidate existing configurations... Not that
big a deal I guess, I had just tried to maintain it with my previous idea(and
this one should result in a shorter config file)
Yes you're right. Changing this will invalidate all existing configurations. I
believe that flume-ng is still considered in alpha state so such change might
be accepted. However it would be nice if one of the committers would agree on
that as well prior coding.
bq. - We need a mechanism to let the selector know a sink has failed. From
there the selector can mark it as dead, rebalancing if necessary. Other than
that the interface looks good to me.
I have an implicit way of doing so in the design. If SinkRunner.chooseSink(int
try) will be executed with parameter try > 1, it means that previously returned
sink has failed and we need another working sink. Consider following example.
Failover sink is keeping track of active sink. It will return this active sink
for call chooseSink(1). However calling chooseSink(2) means that active sink is
not working and we need move to another active sink. However I don't mind
adding explicit method for marking some sink as dead. It was just an idea.
bq. - My biggest concern is how we will test for sink recovery. Most sinks the
only way to know if they are working is to try to attempt to send data through
them
I also found out that most sinks do not return internal state, so that you have
to try them to find actual state. That is actually the reason why I suggested
to put a loop into SinkRunner.PollingRunner.run to keep executing
SinkRunner.chooseSink(int try) until it returns null with increasing try
parameter. This way selector can simply force Runner to try some previously
dead sink and verify that is still dead.
Jarcec
> Modifying SinkRunner to be pluggable to allow for failover/replication.
> -----------------------------------------------------------------------
>
> Key: FLUME-952
> URL: https://issues.apache.org/jira/browse/FLUME-952
> Project: Flume
> Issue Type: Brainstorming
> Components: Sinks+Sources
> Reporter: Juhani Connolly
> Fix For: v1.1.0
>
>
> Implementing the failover sink runner the following was suggested:
> 1. This needs to be implemented on top of FLUME-949 which deals with removing
> the notion of a PollableSink altogether. As a result, the SinkRunner will
> become a concrete implementation that can then allow different sink handling
> policies - such as either a failover policy (needed for this issue), or load
> balancing policy (not needed for this issue). Hence the policy part needs to
> be pluggable rather than the sink runner itself. An example of such a
> construct is the ChannelSelector and ChannelProcessor implementations.
> In Flume-865 I have implemented FailoverSinkRunner as a separate runner, but
> I am open to the idea of making it pluggable if it makes the code more
> maintainable.
> As is, there are many differences between the requirements for Failover and a
> normal Sink runner, including configuration, initialisation, shutdown, error
> handling and event processing. If we were to make this pluggable, many hooks
> would be needed and I don't think there is that much common behavior that
> warrants using a pluggable system rather than just a solid base class.
> - Adding a new sink to a runner, with configuration variables(such as
> priority or weight)
> - Policy for handling process: should this just return a list of sinks to
> process like ChannelSelector and hand off the processing to Process? I think
> that the specific failover policy for each type of runner will be different
> so this feels awkward. I would personally prefer to just pass the process
> call to the pluggable component and let it be responsible for calling process
> on the correct sinks, as well as handling errors.
> Right now I am not convinced for the need to make SinkRunner pluggable, but I
> would be interested to hear other peoples opinions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
