Hi, I'm starting a cluster of kafka brokers using Docker (5 brokers for example, one broker per container). Kafka version 2.12-0.11.0.0, Zookeeper 3.4.10.
*The scenario:* - Starting 1st broker with config below *zoo.cfg* tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/zookeeper/data clientPort=2181 maxClientCnxns=10 minSessionTimeout=4000 maxSessionTimeout=1000000 server.1=0.0.0.0:2888:3888 *server.properties* broker.id=1 listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://broker1_IP:broker1_PORT num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824log.retention.check.interval.ms=300000 zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0 *producer.properties* bootstrap.servers=localhost:9092 compression.type=none *consumer.properties* zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group - Zookeeper is started in standalone mode, then starts kafka - Creating topic /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-test-topic1 - Sending message echo "test_kafka1" | /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-test-topic1 - Checking message /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-test-topic1 --max-messages 1 *Message is recieved* - Describe the topic /opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1 ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 1 Replicas: 1 Isr: 1 - Starting rest 4 brokers *zoo.cfg* on each broker from 1st to 5th (only 0.0.0.0:2888:3888 position differs) tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/zookeeper/data clientPort=2181 maxClientCnxns=10 minSessionTimeout=4000 maxSessionTimeout=1000000 server.1=0.0.0.0:2888:3888 server.2=broker2_IP:broker2_2888:broker2_3888 server.3=broker3_IP:broker3_2888:broker3_3888 server.4=broker4_IP:broker4_2888:broker4_3888 server.5=broker5_IP:broker5_2888:broker5_3888 *server.properties* on each broker from 1st to 5th (broker.id are unique, broker_IP:broker_PORT differs for ech broker) broker.id=N listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://broker_IP:broker_PORT num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824log.retention.check.interval.ms=300000 zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0 *producer.properties* on each broker from 1st to 5th bootstrap.servers=localhost:9092 compression.type=none *consumer.properties* on each broker from 1st to 5th zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group - Restarting zookeeper on each broker to take effect of zoo.cfg - Zookeepers gather into cluster - Topic moved to broker 5 /opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1 ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 5 Replicas: 5 Isr: 5 Is it normal behavior? Or should it stay on broker 1? - Checking message on each broker /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-test-topic1 --max-messages 1 *Message is lost* (message is not lost when topic stays on broker 1, so it is floating situation) Could you help me with this problem. TIA -- *Булат Юсупов */ Инженер-программист / Отдел разработки biusu...@avtodoria.ru / +7 917 889 99 32 skype: usbulat ООО "Автодория" +7 843 524 74 12 Казань, Технопарк в сфере высоких технологий "ИТ-парк", Петербургская, 52, офис 303 www.avtodoria.ru *Инновации спасают жизни!*
