[
https://issues.apache.org/jira/browse/FLUME-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173229#comment-14173229
]
Roshan Naik commented on FLUME-2500:
------------------------------------
I see.. the distributed channel makes for an interesting new use case.
I assume this implies the following topology ? :
app server -> local flume agent -> flume with kafka channel -> destination
That does brings up the question as to why would introduce a flume tier for
such use cases .. instead of directly writing to Kafka.
Basically whatever (a flume agent or something else) is delivering the data to
a flume agent with a Kafka channel, could instead be writing to Kafka directly.
Basically this topology ...
app server -> local flume agent with kafka sink -> kafka -> flume ->
destination
or alternatively
app server -> Kafka -> flume -> destination
Effectively Kafka would function as distributed channel / aggregation layer. Do
you see any disadvantages of these topologies ?
> Add a channel that uses Kafka
> ------------------------------
>
> Key: FLUME-2500
> URL: https://issues.apache.org/jira/browse/FLUME-2500
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Attachments: FLUME-2500.patch
>
>
> Here is the rationale:
> - Kafka does give a HA channel, which means a dead agent does not affect the
> data in the channel - thus reducing delay of delivery.
> - Kafka is used by many companies - it would be a good idea to use Flume to
> pull data from Kafka and write it to HDFS/HBase etc.
> This channel is not going to be useful for cases where Kafka is not already
> used, since it brings is operational overhead of maintaining two systems, but
> if there is Kafka in use - this is good way to integrate Kafka and Flume.
> Here is an a scratch implementation:
> https://github.com/harishreedharan/flume/blob/kafka-channel/flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
