Couple of questions -
1. Which version of Kafka are you using ?
2. Can you please file a JIRA and attach the full producer and broker log
at TRACE level ?
Thanks,
Neha
On Thu, Jul 19, 2012 at 8:12 PM, 刘明敏 <diveintotomor...@gmail.com> wrote:
> In my kafka cluster ,there are 2 brokers,
>
> One of them was killed by accident yesterday, after that ,I tried to
> restart the broker,but seems that it failed to join the cluster.
>
> In zookeeper,it successfully register itself,but after sending logs, no
> data show up in kafka data directory
>
> This is the log of the broker after restart it:
>
> [2012-07-19 17:59:21,154] INFO The number of partitions for topic
> no_appkey : 1 (kafka.utils.Utils$)
> [2012-07-19 17:59:21,156] INFO The number of partitions for topic
> parse_exception : 1 (kafka.utils.Utils$)
> [2012-07-19 17:59:21,160] INFO Starting Kafka server...
> (kafka.server.KafkaServer)
> [2012-07-19 17:59:21,171] INFO starting log cleaner every 60000 ms
> (kafka.log.LogManager)
> [2012-07-19 17:59:21,178] INFO connecting to ZK: XX.XX.XX.XX:2181/kafka
> (kafka.server.KafkaZooKeeper)
> [2012-07-19 17:59:21,187] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2012-07-19 17:59:21,193] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:host.name=mobile-1
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:java.version=1.6.0_17
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:java.vendor=Sun
> Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
> environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
>
> environment:java.class.path=:/home/OUR-SERVER/kafka/libs/jopt-simple-3.2.jar:/home/OUR-SERVER/kafka/libs/log4j-1.2.15.jar:/home/OUR-SERVER/kafka/libs/scala-compiler.jar:/home/OUR-SERVER/kafka/libs/scala-library.jar:/home/OUR-SERVER/kafka/libs/snappy-java-1.0.4.1.jar:/home/OUR-SERVER/kafka/libs/zkclient-0.1.jar:/home/OUR-SERVER/kafka/libs/zookeeper-3.3.4.jar:/home/OUR-SERVER/kafka/kafka-trunk-2a59ad76c6_scala-2.8.0.jar
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
>
> environment:java.library.path=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
> environment:os.version=2.6.32-71.29.1.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client environment:user.name=OUR-SERVER
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
> environment:user.home=/home/OUR-SERVER (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,193] INFO Client
> environment:user.dir=/home/OUR-SERVER/kafka
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,194] INFO Initiating client connection,
> connectString=XX.XX.XX.XX:2181/kafka sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@4a48edb5
> (org.apache.zookeeper.ZooKeeper)
> [2012-07-19 17:59:21,211] INFO Opening socket connection to server
> /XX.XX.XX.XX:2181 (org.apache.zookeeper.ClientCnxn)
> [2012-07-19 17:59:21,215] INFO Socket connection established to
> mobile-1/XX.XX.XX.XX:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2012-07-19 17:59:21,280] INFO Session establishment complete on server
> mobile-1/XX.XX.XX.XX:2181, sessionid = 0x1389e95409a0023, negotiated
> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2012-07-19 17:59:21,282] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2012-07-19 17:59:21,380] INFO Awaiting connections on port 9092
> (kafka.network.Acceptor)
> [2012-07-19 17:59:21,382] INFO Will not load MX4J, mx4j-tools.jar is not in
> the classpath (kafka.utils.Mx4jLoader$)
> [2012-07-19 17:59:21,383] INFO Registering broker /brokers/ids/1
> (kafka.server.KafkaZooKeeper)
> [2012-07-19 17:59:21,404] INFO Registering broker /brokers/ids/1 succeeded
> with id:1,creatorId:XX.XX.XX.XX-1342691961383,host:XX.XX.XX.XX,port:9092
> (kafka.server.KafkaZooKeeper)
> [2012-07-19 17:59:21,417] INFO Starting log flusher every 1000 ms with the
> following overrides Map() (kafka.log.LogManager)
> [2012-07-19 17:59:21,419] INFO Kafka server started.
> (kafka.server.KafkaServer)
>
>
> This is the error from producer:
>
> ERR [20120719-17:53:56.312] exception: Server Exception. : Event queue is
> full of unsent messages, could not send event: {OUR-LOG}
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.async.AsyncProducer.send(AsyncProducer.scala:110)
> ERR [20120719-17:53:56.312] exception:
>
> kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5$$anonfun$apply$2.apply(ProducerPool.scala:129)
> ERR [20120719-17:53:56.312] exception:
>
> kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5$$anonfun$apply$2.apply(ProducerPool.scala:129)
> ERR [20120719-17:53:56.312] exception:
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
> ERR [20120719-17:53:56.312] exception:
> scala.collection.immutable.List.foreach(List.scala:45)
> ERR [20120719-17:53:56.312] exception:
>
> kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5.apply(ProducerPool.scala:129)
> ERR [20120719-17:53:56.312] exception:
>
> kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5.apply(ProducerPool.scala:128)
> ERR [20120719-17:53:56.312] exception:
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
> ERR [20120719-17:53:56.312] exception:
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
> ERR [20120719-17:53:56.312] exception:
>
> kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.scala:128)
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100)
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100)
> ERR [20120719-17:53:56.312] exception:
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
> ERR [20120719-17:53:56.312] exception:
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.ProducerPool.send(ProducerPool.scala:100)
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.Producer.zkSend(Producer.scala:137)
> ERR [20120719-17:53:56.312] exception:
> kafka.producer.Producer.send(Producer.scala:99)
>
> I intentionally shut down the other broker,make the restarted broker the
> only one in the cluster,seems that the producer can not find the broker and
> not sending any events
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu
>
