[
https://issues.apache.org/jira/browse/FLUME-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875545#comment-13875545
]
Ashish Paliwal commented on FLUME-2298:
---------------------------------------
#6 It would be similar to Source/Sink implementation, if we can reuse stable
implementation would be great.
IMHO, can we leave some hook for people to push in their custom distribution
mechanism. Well it's just data replication, so for ex someone with Kafka
cluster might want to reuse it. It may be very tricky to make it generic, it
could be a different DC implementation. Just to record my thoughts. Lets start
with something simple and working, we can refine it later.
One important thing that I feel is DC should be specific to an Agent Name and
Zk path also need to have it.
/flume/distributedchannels/agent1/dc_1 and child nodes can have IP's of master
+ slaves
I am assuming that DC's would be used at only one Layer in Flume Topology to
avoid messages losses, something like Click stream coming in.
> Distributed Channel
> -------------------
>
> Key: FLUME-2298
> URL: https://issues.apache.org/jira/browse/FLUME-2298
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Ted Malaska
> Assignee: Ted Malaska
> Attachments: FlumeDistributedChannelDesign.0.1.pdf
>
>
> This channel will allow for events to be persisted with a plugable method on
> more then one agent or node.
> The goal is to gain the following benefits:
> 1. Events will continue to flow to sinks with out loss or with out large
> delay even in the case of node failure.
> 2. Protect against loss in the case of a complete single node failure
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
