[
https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789801#comment-13789801
]
Jay Kreps commented on KAFKA-369:
---------------------------------
I guess all distributed systems have a problem of bootstrapping knowledge about
the cluster.
In 0.7 the client had a list of hard-coded zk brokers which you connect to, and
from this you can discover the state of the kafka cluster.
There are a number of problems with embedding zookeeper in clients at large
scale:
1. Makes language support hard
2. GC in clients causes havoc
3. Massive zookeeper load
Instead we use the brokers for cluster discovery. As before you need a couple
hardcoded urls to bootstrap metadata from. These can be any three random kafka
brokers (or else use DNS or a vip) this is by no means intended to be a
complete list of your brokers and the producer is not limited in any way to
producing to the brokers it uses to get cluster metadata. These hosts are used
only to find out the state of the cluster the first time. This isn't worse then
the zookeeper situation (you still hardcode a couple urls) but gets rid of all
the other zk issues.
> remove ZK dependency on producer
> --------------------------------
>
> Key: KAFKA-369
> URL: https://issues.apache.org/jira/browse/KAFKA-369
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Fix For: 0.8
>
> Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff,
> kafka_369_v4.diff, kafka_369_v5.diff, kafka_369_v6.diff, kafka_369_v7.diff,
> kafka_369_v8.diff, kafka_369_v9.diff
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, the only place that ZK is actually used is in BrokerPartitionInfo.
> We use ZK to get a list of brokers for making TopicMetadataRequest requests.
> Instead, we can provide a list of brokers in the producer config directly.
> That way, the producer client is no longer dependant on ZK.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
