You can use the partition reassignment tool to move larger partitions from the full node over to the lighter used nodes.
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-Selectivelymovingsomepartitionstoabroker There are also some open source and commercial tools that will rebalance the entire cluster for you automatically and with less time and effort than manually moving around individual partitions. https://www.confluent.io/product/auto-data-balancing/ If you are running a release older than 0.10 then you should be careful to perform any rebalancing in off hours as the replication traffic can negatively impact the production traffic. In the latest releases there is a feature to separately throttle the replication traffic separately from client traffic. -hans > On Jan 6, 2017, at 12:23 PM, R . <[email protected]> wrote: > > Hello, I have a 3node cluster on identical server setups. We've noticed that > one of the kafka's write a lot more data than the other two. Recently, that > kafka server has completely filled our data partition while the other two > kafka servers were still at, for example, 30% capacity. Is there a config > that needs tuning? How do I balance out the servers so that they all more or > less have a similar capacity of storage after processing messages? Thank you > in advance.
