Re: Dynamically adding Kafka brokers

2014-11-04 Thread Neha Narkhede
I agree that KAFKA-1070 would be great to get in. I especially felt the need for something like this while using a few other systems that automated the port, id etc to give a good OOTB experience. Sorry, I lost track of the review. Will do so in the next few days. Thanks, Neha On Mon, Nov 3, 2014

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Jay Kreps
I agree it would be really nice to get KAFKA-1070 figured out. FWIW, the reason for having a name or id other than ip was to make it possible to move the identity to another physical server (e.g. scp the data directory) and have it perform the same role on that new piece of hardware. Systems that

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Gwen Shapira
+1 Thats what we use to generate broker id in automatic deployments. This method makes troubleshooting easier (you know where each broker is running), and doesn't require keeping extra files around. On Mon, Nov 3, 2014 at 2:17 PM, Joe Stein wrote: > Most folks strip the IP and use that as the br

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Joe Stein
Most folks strip the IP and use that as the broker.id. KAFKA-1070 does not yet accommodate for that very widely used method. I think it would be bad if KAFKA-1070 only worked for new installations because that is how people use Kafka today (per https://issues.apache.org/jira/browse/KAFKA-1070?focus

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Joel Koshy
KAFKA-1070 will help with this and is pending a review. On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote: > Hi, > > How do people handle situations, and specifically the broker.id property, > where the Kafka (broker) cluster is not fully defined right away? > > Here's the use cas

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Florian Dambrine
I am automating Kafka clusters in the cloud and I am using Ips (stripped to keep only numbers) to define the broker ids. You might have to add a shell script or whatever fits to set the broker id. I hope this might help a bit. Regards, Le 3 nov. 2014 23:04, "Otis Gospodnetic" a écrit : > Hi, >

Dynamically adding Kafka brokers

2014-11-03 Thread Otis Gospodnetic
Hi, How do people handle situations, and specifically the broker.id property, where the Kafka (broker) cluster is not fully defined right away? Here's the use case we have at Sematext: * Our software ships as a VM * All components run in this single VM, including 1 Kafka broker * Of course, this