Spencer, Kafka (and it's clients) handle failover automatically for you. When you create a topic, you can select a replication factor. For a replication factor n, each partition of the topic will be replicated to n different brokers. At any given time, one of those brokers is considered the leader for that topic and that is the only server you can communicate with to produce new messages. That broker will then make sure the data is copied to the other replicas. If that leader fails, one of the replicas will take over and the producer will have to send data to that node.
But all of this should happen automatically. As long as you set the bootstrap.servers setting (for the new producer) so that at least one of them is always available (ideally it should just include all the brokers), then you shouldn't have to worry about this. Of course if a node fails you have to deal with bringing it back up/moving it to a new machine, but producers should continue to function normally by moving traffic to the new leader. On Fri, Mar 6, 2015 at 9:05 PM, Daniel Moreno <d...@max2.com> wrote: > Hi Spencer, > > You can configure your producers with a list of brokers. You can add all, > but usually at least two of the brokers in your cluster. > > Kind Regards, > > Daniel Moreno > > > On Mar 6, 2015, at 23:43, Spencer Owen <so...@netdocuments.com<mailto: > so...@netdocuments.com>> wrote: > > I've setup a kafka cluster with 3 nodes. > > Which node should I push the data to? I would normally push to kafka01, > but if that node goes down, then the entire cluster goes down. > > How have other people solved this. Maybe a nginx reverse proxy? > > > > http://stackoverflow.com/questions/28911410/which-node-should-i-push-data-to-in-a-cluster > ____________________________________ > This message may contain information that is privileged or confidential. > If you received this transmission in error, please notify the sender by > reply e-mail and delete the message and any attachments. > > Warning: All email sent to this address will be received by the > NetDocuments corporate e-mail system and is subject to archival and review > by someone other than the recipient. > > The services to which this email (or any email in reply or to which this > email replies) relates are provided solely by NetDocuments and NetDocuments > has no responsibility for the services to which this email relates (or any > email in reply or to which this email replies). > -- Thanks, Ewen
