I recently moved away from generating a unique brokerId for each node, in favor of assigning ids in configuration. The reason for this, is that in 0.8, there isn't a convenient way yet to reassign partitions to a new brokerid, should one broker have a failure. So, it seems the only work-around at the moment is to bring up a replacement broker, assign it the same brokerId as one that has failed and is no longer running. The cluster will then automatically replicate all the partitions that were assigned to the failed broker to the new broker.
This appears the only operational way to deal with failed brokers, at the moment. Longer term, it would be great if the cluster were self-healing, and if a broker went down, we could mark it as no longer available somehow, and the cluster would then reassign and re-replicate partitions to new brokers, that were previously assigned to the failed broker. I expect something like this will be available in future versions, but that doesn't appear the case at present. And related, it would be nice, in the interests of horizontal scalability, to have an easy way for the cluster to dynamically rebalance load, if new nodes are added to the cluster (or to at least prefer assigning new partitions to brokers which have more space available). I expect this will be something to prioritize in the future versions as well. Jason On Wed, Oct 2, 2013 at 1:00 PM, Sriram Subramanian < srsubraman...@linkedin.com> wrote: > I agree that we need a unique id and have something independent of the > machine. I am not sure you want a dependency on ZK to generate the unique > id though. There are other ways to generate an unique id (Example - UUID). > In case there was a collision (highly unlikely), the node creation in ZK > will anyways fail and the broker can regenerate another id. > > On 10/2/13 9:52 AM, "Jay Kreps" <jay.kr...@gmail.com> wrote: > > >There are scenarios in which you want a hostname to change or you want to > >move the stored data off one machine onto another. This is the motivation > >systems have for having a layer of indirection between the location and > >the > >identity of the nodes. > > > >-Jay > > > > > >On Wed, Oct 2, 2013 at 9:23 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > > >> Wondering what is the reason behind decoupling the node id with its > >> physical host(port)? If we found that for example, node 1 is not owning > >>any > >> partitions, how would we know which physical machine is this node then? > >> > >> Guozhang > >> > >> > >> On Wed, Oct 2, 2013 at 9:07 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> > >> > I'm in favor of doing this if someone is willing to work on it! I > >>agree > >> it > >> > would really help with easy provisioning. > >> > > >> > I filed a bug to discuss and track: > >> > https://issues.apache.org/jira/browse/KAFKA-1070 > >> > > >> > Some comments: > >> > 1. I'm not in favor of having a pluggable strategy, unless we are > >>really > >> > really sure this is an area where people are going to get a lot of > >>value > >> by > >> > writing lots of plugins. I am not at all sure why you would want to > >> retain > >> > the current behavior if you had a good strategy for automatically > >> > generating ids. Basically plugins are an evil we only want to accept > >>when > >> > either we don't understand the problem or the solutions have such > >>extreme > >> > tradeoffs that there is no single "good approach". Plugins cause > >>problems > >> > for upgrades, testing, documentation, user understandability, code > >> > understandability, etc. > >> > 2. The node id can't be the host or port or anything tied to the > >>physical > >> > machine or its location on the network because you need to be able to > >> > change these things. I recommend we just keep an integer. > >> > > >> > -Jay > >> > > >> > > >> > On Tue, Oct 1, 2013 at 7:08 AM, Aniket Bhatnagar < > >> > aniket.bhatna...@gmail.com > >> > > wrote: > >> > > >> > > Right. It is currently java integer. However, as per previous > >>thread, > >> it > >> > > seems possible to change it to a string. In that case, we can use > >> > instance > >> > > IDs, IP addresses, custom ID generators, etc. > >> > > How are you currently generating broker IDs from IP address? Chef > >> script > >> > or > >> > > custom shell script? > >> > > > >> > > > >> > > On 1 October 2013 18:34, Maxime Brugidou <maxime.brugi...@gmail.com > > > >> > > wrote: > >> > > > >> > > > I think it currently is a java (signed) integer or maybe this was > >> > > > zookeeper? > >> > > > We are generating the id from IP address for now but this is not > >> ideal > >> > > (and > >> > > > can cause integer overflow with java signed ints) > >> > > > On Oct 1, 2013 12:52 PM, "Aniket Bhatnagar" < > >> > aniket.bhatna...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > I would like to revive an older thread around auto generating > >> broker > >> > > ID. > >> > > > As > >> > > > > a AWS user, I would like Kafka to just use the instance's ID or > >> > > > instance's > >> > > > > IP or instance's internal domain (whichever is easier). This > >>would > >> > > mean I > >> > > > > can easily clone from a AMI to launch kafka instances without > >> having > >> > to > >> > > > > worry about setting a unique broker ID. This also alows me to > >>setup > >> > > auto > >> > > > > scaling. > >> > > > > > >> > > > > I realize 1 size may not fit all in this case. Other strategies > >> that > >> > > may > >> > > > > work for other cloud providers are generate the UUID and > >>persist it > >> > on > >> > > a > >> > > > > disk, etc. > >> > > > > > >> > > > > What I propose is a way to define a a broker ID generation > >>strategy > >> > in > >> > > > the > >> > > > > configuration file which points to a class file that is > >>responsible > >> > for > >> > > > > generating the ID. Is this something being already worked upon? > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> > >
