[
https://issues.apache.org/jira/browse/FLUME-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199594#comment-13199594
]
[email protected] commented on FLUME-865:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3750/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
This includes the changes from FLUME-945 and is thus subject to change with it.
I have a separate patch for the diff from 945->865 if someone wants it.
A failover sink runner is added, along with a SinkRunnerFactory. The
configuration reading is modified so as to read the runner configuration and
pass it to the SinkRunnerFactory.
A failover sink is used by setting the runner type to failover for every
participating sink. In addition each sink assigned to a runner must set the
same runner.name and a unique runner.priority
The runner will poll only the highest priority live sink. Should it fail by
throwing EventDeliveryException, it will throw it into a pile of dead sinks.
Only once all sinks are exhausted are the dead sinks revived.
Some issues that are outstanding
- Not all sinks throw EventDeliveryException, or have a clear time when they
should be considered "dead". Coupling such sinks with the failover runner will
not result in failover
- Sinks do not have a clear mechanism to test for their liveliness, other than
perhaps polling LifecycleState. However for most sinks lifecyclestate tends to
remain in START even if the sink is unable to do anything(e.g. avro sink
failing to connect)
This addresses bug FLUME-865.
https://issues.apache.org/jira/browse/FLUME-865
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/PollableSink.java e86cc59
flume-ng-core/src/main/java/org/apache/flume/Sink.java ab9b63c
flume-ng-core/src/main/java/org/apache/flume/SinkRunner.java 7f5cb17
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 90b8a86
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkRunner.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java 7c87bf7
flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java 5722cd1
flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java
afcf1c3
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
47addd1
flume-ng-core/src/main/java/org/apache/flume/sink/SinkRunnerFactory.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 9718491
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkRunner.java
PRE-CREATION
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
1ee1f8e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
bea0a3c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
f1e23bf
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
716bbf5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
ff20beb
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java
5fc3943
Diff: https://reviews.apache.org/r/3750/diff
Testing
-------
All unit tests pass.
A new unit test was added. It creates a memory channel and a failover runner
with 3 sinks that consume a preset number of events then send
EventDeliveryException. Events are fed to the channel and assertions are made
that the runner is failing over to the correct sinks. This test also passes.
I also tried to run this on a real cluster, with one master source using a
failover runner to three avro sinks which would each fed into separate agents
which I would kill off to test the failover. Unfortunately because of AvroSink
not throwing EventDeliveryException, this test could not be completed
succesfully, and I felt it was beyond the scope of this ticket to modify
AvroSink
Thanks,
Juhani
> Implement failover sink
> ------------------------
>
> Key: FLUME-865
> URL: https://issues.apache.org/jira/browse/FLUME-865
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Assignee: Juhani Connolly
> Fix For: v1.1.0
>
> Attachments: Flume-865.patch
>
>
> It would be nice if the flume-ng would have ability to failover to different
> sink in case that the active one is not responding (e.g. before failing the
> transaction).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
