kafka git commit: DOC: Documentation for Throttled Replication

[gwenshap]

Repository: kafka
Updated Branches:
  refs/heads/0.10.1 d273480c4 -> b729ac520


DOC: Documentation for Throttled Replication

Author: Ben Stopford 

Reviewers: Jun Rao, Gwen Shapira

Closes #2033 from benstopford/throttling-docs

(cherry picked from commit b8cfa167edfefbae52bec98eb357121e6437cef6)
Signed-off-by: Gwen Shapira 


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/b729ac52
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/b729ac52
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/b729ac52

Branch: refs/heads/0.10.1
Commit: b729ac520605b5f8eec5bc23d8625891a3709547
Parents: d273480
Author: Ben Stopford 
Authored: Tue Oct 18 14:24:52 2016 -0700
Committer: Gwen Shapira 
Committed: Tue Oct 18 14:25:03 2016 -0700

--
 docs/ops.html | 94 ++
 1 file changed, 94 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kafka/blob/b729ac52/docs/ops.html
--
diff --git a/docs/ops.html b/docs/ops.html
index ed0c153..c26f0cb 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -346,6 +346,100 @@ Topic:foo PartitionCount:1ReplicationFactor:3 
Configs:
Topic: foo  Partition: 0Leader: 5   Replicas: 5,6,7 Isr: 
5,6,7
 
 
+Limiting Bandwidth Usage during 
Data Migration
+Kafka lets you apply a throttle to replication traffic, setting an upper bound 
on the bandwidth used to move replicas from machine to machine. This is useful 
when rebalancing a cluster, bootstrapping a new broker or adding or removing 
brokers, as it limits the impact these data-intensive operations will have on 
users.
+
+There are two interfaces that can be used to engage a throttle. The simplest, 
and safest, is to apply a throttle when invoking the 
kafka-reassign-partitions.sh, but kafka-configs.sh can also be used to view and 
alter the throttle values directly.
+
+So for example, if you were to execute a rebalance, with the below command, it 
would move partitions at no more than 50MB/s.
+$ bin/kafka-reassign-partitions.sh --zookeeper myhost:2181--execute 
--reassignment-json-file bigger-cluster.json —throttle 5000
+When you execute this script you will see the throttle engage:
+
+The throttle limit was set to 5000 B/s
+Successfully started reassignment of partitions.
+Should you wish to alter the throttle, during a rebalance, say to increase 
the throughput so it completes quicker, you can do this by re-running the 
execute command passing the same reassignment-json-file:
+$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --execute 
--reassignment-json-file bigger-cluster.json --throttle 7
+There is an existing assignment running.
+The throttle limit was set to 7 B/s
+
+Once the rebalance completes the administrator can check the status of the 
rebalance using the --verify option.
+If the rebalance has completed, the throttle will be removed via the 
--verify command. It is important that
+administrators remove the throttle in a timely manner once rebalancing 
completes by running the command with
+the --verify option. Failure to do so could cause regular replication 
traffic to be throttled. 
+When the --verify option is executed, and the reassignment has completed, 
the script will confirm that the throttle was removed:
+
+$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --verify 
--reassignment-json-file bigger-cluster.json
+Status of partition reassignment:
+Reassignment of partition [my-topic,1] completed successfully
+Reassignment of partition [mytopic,0] completed successfully
+Throttle was removed.
+
+The administrator can also validate the assigned configs using the 
kafka-configs.sh. There are two pairs of throttle
+configuration used to manage the throttling process. The throttle value 
itself. This is configured, at a broker
+level, using the dynamic properties: 
+
+leader.replication.throttled.rate
+follower.replication.throttled.rate
+
+There is also an enumerated set of throttled replicas: 
+
+leader.replication.throttled.replicas
+follower.replication.throttled.replicas
+
+Which are configured per topic. All four config values are automatically 
assigned by kafka-reassign-partitions.sh
+(discussed below). 
+To view the throttle limit configuration:
+
+$ bin/kafka-configs.sh --describe --zookeeper localhost:2181 
--entity-type brokers
+Configs for brokers '2' are 
leader.replication.throttled.rate=7,follower.replication.throttled.rate=7
+Configs for brokers '1' are 
leader.replication.throttled.rate=7,follower.replication.throttled.rate=7
+
+This shows the throttle applied 

kafka git commit: DOC: Documentation for Throttled Replication

[gwenshap]

Repository: kafka
Updated Branches:
  refs/heads/trunk 3c02e5a20 -> b8cfa167e


DOC: Documentation for Throttled Replication

Author: Ben Stopford 

Reviewers: Jun Rao, Gwen Shapira

Closes #2033 from benstopford/throttling-docs


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/b8cfa167
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/b8cfa167
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/b8cfa167

Branch: refs/heads/trunk
Commit: b8cfa167edfefbae52bec98eb357121e6437cef6
Parents: 3c02e5a
Author: Ben Stopford 
Authored: Tue Oct 18 14:24:52 2016 -0700
Committer: Gwen Shapira 
Committed: Tue Oct 18 14:24:52 2016 -0700

--
 docs/ops.html | 94 ++
 1 file changed, 94 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kafka/blob/b8cfa167/docs/ops.html
--
diff --git a/docs/ops.html b/docs/ops.html
index ed0c153..c26f0cb 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -346,6 +346,100 @@ Topic:foo PartitionCount:1ReplicationFactor:3 
Configs:
Topic: foo  Partition: 0Leader: 5   Replicas: 5,6,7 Isr: 
5,6,7
 
 
+Limiting Bandwidth Usage during 
Data Migration
+Kafka lets you apply a throttle to replication traffic, setting an upper bound 
on the bandwidth used to move replicas from machine to machine. This is useful 
when rebalancing a cluster, bootstrapping a new broker or adding or removing 
brokers, as it limits the impact these data-intensive operations will have on 
users.
+
+There are two interfaces that can be used to engage a throttle. The simplest, 
and safest, is to apply a throttle when invoking the 
kafka-reassign-partitions.sh, but kafka-configs.sh can also be used to view and 
alter the throttle values directly.
+
+So for example, if you were to execute a rebalance, with the below command, it 
would move partitions at no more than 50MB/s.
+$ bin/kafka-reassign-partitions.sh --zookeeper myhost:2181--execute 
--reassignment-json-file bigger-cluster.json —throttle 5000
+When you execute this script you will see the throttle engage:
+
+The throttle limit was set to 5000 B/s
+Successfully started reassignment of partitions.
+Should you wish to alter the throttle, during a rebalance, say to increase 
the throughput so it completes quicker, you can do this by re-running the 
execute command passing the same reassignment-json-file:
+$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --execute 
--reassignment-json-file bigger-cluster.json --throttle 7
+There is an existing assignment running.
+The throttle limit was set to 7 B/s
+
+Once the rebalance completes the administrator can check the status of the 
rebalance using the --verify option.
+If the rebalance has completed, the throttle will be removed via the 
--verify command. It is important that
+administrators remove the throttle in a timely manner once rebalancing 
completes by running the command with
+the --verify option. Failure to do so could cause regular replication 
traffic to be throttled. 
+When the --verify option is executed, and the reassignment has completed, 
the script will confirm that the throttle was removed:
+
+$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --verify 
--reassignment-json-file bigger-cluster.json
+Status of partition reassignment:
+Reassignment of partition [my-topic,1] completed successfully
+Reassignment of partition [mytopic,0] completed successfully
+Throttle was removed.
+
+The administrator can also validate the assigned configs using the 
kafka-configs.sh. There are two pairs of throttle
+configuration used to manage the throttling process. The throttle value 
itself. This is configured, at a broker
+level, using the dynamic properties: 
+
+leader.replication.throttled.rate
+follower.replication.throttled.rate
+
+There is also an enumerated set of throttled replicas: 
+
+leader.replication.throttled.replicas
+follower.replication.throttled.replicas
+
+Which are configured per topic. All four config values are automatically 
assigned by kafka-reassign-partitions.sh
+(discussed below). 
+To view the throttle limit configuration:
+
+$ bin/kafka-configs.sh --describe --zookeeper localhost:2181 
--entity-type brokers
+Configs for brokers '2' are 
leader.replication.throttled.rate=7,follower.replication.throttled.rate=7
+Configs for brokers '1' are 
leader.replication.throttled.rate=7,follower.replication.throttled.rate=7
+
+This shows the throttle applied to both leader and follower side of the 
replication protocol. By default both sides
+are assigned the same