Hello Alex, ZooKeeper nodes that Kafka brokers create in /brokers/ids path, upon registering themselves that they are available, are ephemeral, so not persistent. Ephemeral ZooKeeper nodes live as long as ZooKeeper session that created them is active. The /broker/ids child nodes are gone by design when broker to ZooKeeper connection is down long enough (e.g. due to network partition, or broker actually stopped/crashed). They are meant to signal availability of particular broker in the Kafka cluster. Broker announces its availability so cluster members can be discovered dynamically (as brokers join or leave the cluster) by other Kafka brokers in the cluster and by Kafka clients. So that data is meant to be lost by design.
Maybe ZooKeeper node type could be mentioned in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper Kind regards, Stevo Slavic. On Wed, Dec 28, 2016 at 10:57 AM, Alex Eftimie < alex.efti...@getyourguide.com> wrote: > Quick update: digging into Zookeeper data, we observed that the > /brokers/ids path was empty. Restarting the kafka nodes repopulated > zookeeper data, and now the error is gone (we are able to create new > topics). > > We didn’t alter data in Zookeeper manually, but recently we added 2 nodes > to a 3 nodes zookeeper cluster. Also, we had a network partitioning issue > in the past. Could any of these be the reason for the lost ZK data? > > Any other hints on what may have generated the data loss in ZK? > > Thanks, > Alex > > > On 28 Dec 2016, at 10:00, Alex Eftimie <alex.efti...@getyourguide.com> > wrote: > > > > Hello, > > > > We recently migrated from Kafka 0.8 to 0.10, while keeping the log > format and internal communication at 0.9 version[1]. We have a cluster of > two nodes which is working correctly for 10 topics (producers/consumers > work fine). > > > > Trying to create a new topic raises: > > kafka-topics.sh --zookeeper zknode:2181 --create > --replication-factor 1 --partitions 2 --topic newtopic > > Error while executing topic command : replication factor: 1 larger > than available brokers: 0 > > [2016-12-28 09:30:02,177] ERROR org.apache.kafka.common.errors. > InvalidReplicationFactorException: replication factor: 1 larger than > available brokers: 0 > > (kafka.admin.TopicCommand$) > > > > There are two available brokers, so this must be a different error > hidden behind this error message. There’s nothing in the log file. > > > > Where should we start investigating? > > > > Thanks, > > > > Alex Eftimie > > Software Engineer DevOps > > > > [1] > > kafka version: 0.10.1.0 > > # config: > > inter.broker.protocol.version=0.9.0.1 > > log.message.format.version=0.9.0.1 > > > > > -- > > > ------------------------------ > > [image: email GYG_Logo__Standart_Transparent_400x600_WEB copy.png] > <http://www.getyourguide.com> > > GetYourGuide AG > > Technoparkstrasse 1 > > 8005 Zürich > > Switzerland > > <https://www.facebook.com/GetYourGuide> <https://twitter.com/GetYourGuide> > <https://www.instagram.com/getyourguide/> > <https://www.linkedin.com/company/getyourguide-ag> [image: Icon_4.png] > <http://www.getyourguide.com> >
