[
https://issues.apache.org/jira/browse/KAFKA-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903779#comment-13903779
]
Jun Rao commented on KAFKA-1182:
--------------------------------
One way to fix this issue is the following. In addition to register live
brokers in ZK, also register brokers ever assigned to a cluster. During topic
creation, we will require that assigned (not live) brokers be more or equal to
the replication factor. If so, the topic will be created. The topic may be
under replicated or unavailable if one or more assigned brokers are down.
Making this fully work will need more thoughts. At the minimum, we need to
1. Change how topic creation works.
2. We need to have a way to de-assign a broker now.
3. We have to think though how this affect admin tools like partition
reassignment.
4. We need to think about how to make it a backward compatible change.
So, this is potentially a non-trivial change and I recommend that we defer this
to post 0.9.
Hanish,
What's the replication factor that you plan to use? As a temporary solution,
could you just allocate one or two more brokers in a cluster to make sure there
are always enough brokers for topic creation in the presence of broker failure?
> Topic not created if number of live brokers less than # replicas
> ----------------------------------------------------------------
>
> Key: KAFKA-1182
> URL: https://issues.apache.org/jira/browse/KAFKA-1182
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 0.8.0
> Environment: Centos 6.3
> Reporter: Hanish Bansal
> Assignee: Jun Rao
>
> We are having kafka cluster of 2 nodes. (Using Kafka 0.8.0 version)
> Replication Factor: 2
> Number of partitions: 2
> Actual Behaviour:
> Out of two nodes, if any one node goes down then topic is not created in
> kafka.
> Steps to Reproduce:
> 1. Create a 2 node kafka cluster with replication factor 2
> 2. Start the Kafka cluster
> 3. Kill any one node
> 4. Start the producer to write on a new topic
> 5. Observe the exception stated below:
> 2013-12-12 19:37:19 0 [WARN ] ClientUtils$ - Fetching topic metadata with
> correlation id 3 for topics [Set(test-topic)] from broker
> [id:0,host:122.98.12.11,port:9092] failed
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:49)
> at
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:186)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:150)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> at
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:149)
> at
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:95)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:76)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> Expected Behaviour:
> In case of live brokers less than # replicas:
> There should be topic created so at least live brokers can receive the data.
> They can replicate data to other broker once any down broker comes up.
> Because now in case of live brokers less than # replicas, there is complete
> loss of data.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
