[
https://issues.apache.org/jira/browse/FLUME-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970000#comment-15970000
]
Jeff Holoman commented on FLUME-3086:
-------------------------------------
I don't think I was clear enough earlier. The channel doesn't rely on zookeeper
for offset storage. Additionally, we must commit every transaction due to our
delivery guarantees. We don't commit each message. Higher batch sizes will lead
to less frequent commits. We've seen excellent throughout on the channel in
production deployments.
Perhaps you could post your config on the Dev or user mailing list.
> KafkaChannel commits consume offset to Zookeeper every time causing
> performance problem
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-3086
> URL: https://issues.apache.org/jira/browse/FLUME-3086
> Project: Flume
> Issue Type: Bug
> Components: Kafka Channel
> Affects Versions: 1.7.0
> Reporter: JiYeon OH
>
> Currently, the AUTO_COMMIT_ENABLED setting is hard corded to
> false(KafkaChannel.java line 285), and the commit is done every
> transaction(KafkaChannel.java line 567). This will cause large overhead to
> Kafka Zookeeper when consuming very large amount of data because every
> consumer in every node will be writing to Zookeeper.
> I actually had this problem and affected other running consumers' offsets on
> the Zookeeper. So I suggest to change the hard coded setting
> AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value,
> and remove the commitOffsets() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
