[
https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105639#comment-15105639
]
Eno Thereska commented on KAFKA-1464:
-------------------------------------
An alternative to throttling background maintenance traffic is to use a
priority levels (just two: foreground and background). This has the advantage
of being fairly simple and allows for important replication work to proceed
fast if there is little or no foreground traffic. If most of the contention
happens at the disk (as [~mjuarez] seems to indicate) then priorities
implemented as two queues at the receiving end could be sufficient. However, if
the network is a problem as well, then throttling would probably work best
since it limits background traffic at the source.
> Add a throttling option to the Kafka replication tool
> -----------------------------------------------------
>
> Key: KAFKA-1464
> URL: https://issues.apache.org/jira/browse/KAFKA-1464
> Project: Kafka
> Issue Type: New Feature
> Components: replication
> Affects Versions: 0.8.0
> Reporter: mjuarez
> Assignee: Ismael Juma
> Priority: Minor
> Labels: replication, replication-tools
> Fix For: 0.9.1.0
>
>
> When performing replication on new nodes of a Kafka cluster, the replication
> process will use all available resources to replicate as fast as possible.
> This causes performance issues (mostly disk IO and sometimes network
> bandwidth) when doing this in a production environment, in which you're
> trying to serve downstream applications, at the same time you're performing
> maintenance on the Kafka cluster.
> An option to throttle the replication to a specific rate (in either MB/s or
> activities/second) would help production systems to better handle maintenance
> tasks while still serving downstream applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
