[
https://issues.apache.org/jira/browse/FLUME-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14492603#comment-14492603
]
Tao Li commented on FLUME-2500:
-------------------------------
I have a question about this issue.
Flume support transaction between source and channel. So for kafka channel:
1. If we use "sync" kafka producer, it surely can guarantee the transaction,
but sync send has low performance.
2. If we use "async" kafka producer, the performance will be better, but it
can't guarantee the transaction.
How do you think on it?
> Add a channel that uses Kafka
> ------------------------------
>
> Key: FLUME-2500
> URL: https://issues.apache.org/jira/browse/FLUME-2500
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2500-1.patch, FLUME-2500-2.patch,
> FLUME-2500.patch, FLUME-2500.patch
>
>
> Here is the rationale:
> - Kafka does give a HA channel, which means a dead agent does not affect the
> data in the channel - thus reducing delay of delivery.
> - Kafka is used by many companies - it would be a good idea to use Flume to
> pull data from Kafka and write it to HDFS/HBase etc.
> This channel is not going to be useful for cases where Kafka is not already
> used, since it brings is operational overhead of maintaining two systems, but
> if there is Kafka in use - this is good way to integrate Kafka and Flume.
> Here is an a scratch implementation:
> https://github.com/harishreedharan/flume/blob/kafka-channel/flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
