In my kafka cluster ,there are 2 brokers,
One of them was killed by accident yesterday, after that ,I tried to
restart the broker,but seems that it failed to join the cluster.
In zookeeper,it successfully register itself,but after sending logs, no
data show up in kafka data directory
This is the log of the broker after restart it:
[2012-07-19 17:59:21,154] INFO The number of partitions for topic
no_appkey : 1 (kafka.utils.Utils$)
[2012-07-19 17:59:21,156] INFO The number of partitions for topic
parse_exception : 1 (kafka.utils.Utils$)
[2012-07-19 17:59:21,160] INFO Starting Kafka server...
(kafka.server.KafkaServer)
[2012-07-19 17:59:21,171] INFO starting log cleaner every 60000 ms
(kafka.log.LogManager)
[2012-07-19 17:59:21,178] INFO connecting to ZK: XX.XX.XX.XX:2181/kafka
(kafka.server.KafkaZooKeeper)
[2012-07-19 17:59:21,187] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2012-07-19 17:59:21,193] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:host.name=mobile-1
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:java.version=1.6.0_17
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:java.vendor=Sun
Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:java.class.path=:/home/OUR-SERVER/kafka/libs/jopt-simple-3.2.jar:/home/OUR-SERVER/kafka/libs/log4j-1.2.15.jar:/home/OUR-SERVER/kafka/libs/scala-compiler.jar:/home/OUR-SERVER/kafka/libs/scala-library.jar:/home/OUR-SERVER/kafka/libs/snappy-java-1.0.4.1.jar:/home/OUR-SERVER/kafka/libs/zkclient-0.1.jar:/home/OUR-SERVER/kafka/libs/zookeeper-3.3.4.jar:/home/OUR-SERVER/kafka/kafka-trunk-2a59ad76c6_scala-2.8.0.jar
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:java.library.path=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:os.version=2.6.32-71.29.1.el6.x86_64
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client environment:user.name=OUR-SERVER
(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:user.home=/home/OUR-SERVER (org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,193] INFO Client
environment:user.dir=/home/OUR-SERVER/kafka (org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,194] INFO Initiating client connection,
connectString=XX.XX.XX.XX:2181/kafka sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@4a48edb5(org.apache.zookeeper.ZooKeeper)
[2012-07-19 17:59:21,211] INFO Opening socket connection to server
/XX.XX.XX.XX:2181 (org.apache.zookeeper.ClientCnxn)
[2012-07-19 17:59:21,215] INFO Socket connection established to
mobile-1/XX.XX.XX.XX:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2012-07-19 17:59:21,280] INFO Session establishment complete on server
mobile-1/XX.XX.XX.XX:2181, sessionid = 0x1389e95409a0023, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2012-07-19 17:59:21,282] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2012-07-19 17:59:21,380] INFO Awaiting connections on port 9092
(kafka.network.Acceptor)
[2012-07-19 17:59:21,382] INFO Will not load MX4J, mx4j-tools.jar is not in
the classpath (kafka.utils.Mx4jLoader$)
[2012-07-19 17:59:21,383] INFO Registering broker /brokers/ids/1
(kafka.server.KafkaZooKeeper)
[2012-07-19 17:59:21,404] INFO Registering broker /brokers/ids/1 succeeded
with id:1,creatorId:XX.XX.XX.XX-1342691961383,host:XX.XX.XX.XX,port:9092
(kafka.server.KafkaZooKeeper)
[2012-07-19 17:59:21,417] INFO Starting log flusher every 1000 ms with the
following overrides Map() (kafka.log.LogManager)
[2012-07-19 17:59:21,419] INFO Kafka server started.
(kafka.server.KafkaServer)
This is the error from producer:
ERR [20120719-17:53:56.312] exception: Server Exception. : Event queue is
full of unsent messages, could not send event: {OUR-LOG}
ERR [20120719-17:53:56.312] exception:
kafka.producer.async.AsyncProducer.send(AsyncProducer.scala:110)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5$$anonfun$apply$2.apply(ProducerPool.scala:129)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5$$anonfun$apply$2.apply(ProducerPool.scala:129)
ERR [20120719-17:53:56.312] exception:
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
ERR [20120719-17:53:56.312] exception:
scala.collection.immutable.List.foreach(List.scala:45)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5.apply(ProducerPool.scala:129)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1$$anonfun$apply$mcVI$sp$5.apply(ProducerPool.scala:128)
ERR [20120719-17:53:56.312] exception:
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
ERR [20120719-17:53:56.312] exception:
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.scala:128)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100)
ERR [20120719-17:53:56.312] exception:
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
ERR [20120719-17:53:56.312] exception:
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
ERR [20120719-17:53:56.312] exception:
kafka.producer.ProducerPool.send(ProducerPool.scala:100)
ERR [20120719-17:53:56.312] exception:
kafka.producer.Producer.zkSend(Producer.scala:137)
ERR [20120719-17:53:56.312] exception:
kafka.producer.Producer.send(Producer.scala:99)
I intentionally shut down the other broker,make the restarted broker the
only one in the cluster,seems that the producer can not find the broker and
not sending any events
--
Best Regards
----------------------
刘明敏 | mmLiu
