I'm copying the Kafka user list here: > We have a 3 node zookeeper cluster and a kafka cluster (3 nodes) using this > zookeeper cluster. we want to migrate the zookeeper nodes to better boxes ( > hardware improvements). we already setup 3 new nodes. > > can some one tell me what is the safe way to do the migration. > > 1. what happens if we change the zookeeper connection string in kafka nodes > to point to 3 new ZK nodes and start the kafka nodes one by one?
The new cluster will start with an empty state. Switching to a new cluster abruptly like you suggest will mean loss of data, which is bad. Also, you won't be able to switch all brokers at once, so for some time, you'll have brokers talking to different ensembles, which can lead to an inconsistent state as well. > 2. Do we need to change the old and new zookeeper nodes configuration before > applying any changes on kafka nodes (zk clients)? like adding new nodes as > observers etc... You need to introduce one new node at a time. In the same step that you introduce a node, you remove a node from the old ensemble. To do this, you need to change the server configuration at each step to remove a server and add a server, and wait until a quorum follows a leader before moving to the next step. This way you guarantee that the state of zookeeper in the old ensemble migrates to the new ensemble. > 3. what happens if we change the kafka configuration to talk to all 6 ZK > Nodes and later shut down 3 old nodes and reconfigure the kafka to talk to 3 > new ZK nodes. You can get into an inconsistent state if you do this. There is a chance that it works, but I don't recommend it. -Flavio > > > > -- > View this message in context: > http://zookeeper-user.578899.n2.nabble.com/zookeeper-cluster-change-impact-on-kafka-tp7581711.html > Sent from the zookeeper-user mailing list archive at Nabble.com.
