Hello Guys,
I've been working on setting up a multi node Kafka deployment and need help on it. I've been able to set up a 3 node cluster but am running into issues (performance and getting errors too) every few hours. I've attached the steps that I used to set up the cluster. Can someone please take a look at it and let me know if I'm missing anything? I know this is a pretty vague request but I've tried many other options and they're not working for me. It'll be perfectly fine if you can also direct me to another source or if you can provide setup instructions that have worked for you in the past. Sorry for the trouble and thanks in advance. Thanks, Dipan Shah
Ref Links : https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04 https://github.com/airbnb/kafkat Create a User for Kafka Create a user called kafka using the useradd command sudo useradd kafka -m Set its password using passwd: sudo passwd kafka Add it to the sudo group so that it has the privileges required to install Kafka's dependencies. This can be done using the adduser command: sudo adduser kafka sudo Your Kafka user is now ready. Log into it using su: su - kafka Install Java sudo apt-get update sudo apt-get install default-jre Install ZooKeeper sudo apt-get install zookeeperd Download and Extract Kafka Binaries mkdir -p ~/Downloads wget "http://mirror.cc.columbia.edu/pub/software/apache/kafka/0.10.2.1/kafka_2.10-0.10.2.1.tgz"; -O ~/Downloads/kafka.tgz mkdir -p ~/kafka && cd ~/kafka tar -xvzf ~/Downloads/kafka.tgz --strip 1 Edit server properties sudo nano ~/kafka/config/server.properties Change broker.id=0 to unique server number. Uncomment delete.topic.enable=true Change - zookeeper.connect=192.168.0.172:2181,192.168.0.173:2181,192.168.0.174:2181 Edit Zookeeper properties sudo nano ~/kafka/config/zookeeper.properties Change - dataDir=/var/lib/zookeeper Add # added servers for cluster server.1=192.168.0.172:2888:3888 server.2=192.168.0.173:2888:3888 server.3=192.168.0.174:2888:3888 initLimit=5 syncLimit=2 To change folder owner sudo chown -R kafka:kafka /var/lib/zookeeper Create Myid file sudo nano /var/lib/zookeeper/myid Write the server id number here. For example in server 192.168.0.63. Open file and write mere 1. To open port sudo iptables -I INPUT -p tcp -m tcp --dport 2181 -j ACCEPT sudo iptables -I INPUT -p tcp -m tcp --dport 3888 -j ACCEPT sudo iptables -I INPUT -p tcp -m tcp --dport 2888 -j ACCEPT To Start Zookeeper In Deamon bin/zookeeper-server-start.sh -daemon config/zookeeper.properties In console (Dont use this. This is just for debugging) bin/zookeeper-server-start.sh config/zookeeper.properties To Stop Zookeepe In Deamon bin/zookeeper-server-stop.sh -daemon config/zookeeper.properties In console bin/zookeeper-server-stop.sh config/zookeeper.properties In Service sudo service zookeeper stop To check Zooker is running: sudo service zookeeper status Find Zookeper process ps -ef | grep -i 'zookeeper' Kill the process sudo kill -9 <pid> To Start Kafka In Deamon bin/kafka-server-start.sh -daemon config/server.properties For Datadog JMX_PORT=3800 ./kafka-server-start.sh -daemon ../config/server.properties In Console (Dont use this. This is just for debugging) bin/kafka-server-start.sh config/server.properties To Stop bin/kafka-server-stop.sh config/server.properties To create topic bin/kafka-topics.sh --create --zookeeper 192.168.0.172:2181 --replication-factor 3 --partitions 3 --topic test2 To describe bin/kafka-topics.sh --describe --zookeeper 192.168.0.63:2181 --topic ishita To publish bin/kafka-console-producer.sh --broker-list 192.168.0.63:9092 --topic test2 To consume bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.63:9092 --from-beginning --topic ishita To consume with group bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.63:9092 --topic ishita --consumer-property group.id=ishita-group To get list of Topics bin/kafka-topics.sh --list --zookeeper 192.168.0.172:2181 To delete Topic bin/kafka-topics.sh --zookeeper 192.168.0.172:2181 --delete --topic test Producer Language : Python Framework : Flask ps -ef | grep -i 'python' Python App: export FLASK_APP=hello.py flask run -h 192.168.0.63 & exit Help Links https://www.youtube.com/watch?v=t2pN4_HfGbY&feature=youtu.be
