Re: kafka topic not exist after creation with api

2015-01-23 Thread Jun Rao 
Are you on the latest 0.8.2 branch? We did fix KAFKA-1738 a couple of
months ago which could prevent new topics from created.

Thanks,

Jun

On Thu, Jan 22, 2015 at 9:44 PM, Manu Zhang owenzhang1...@gmail.com wrote:

 Hi all ,
 My application creates kafka topic at runtime with AdminUtils.createTopic
 but that topic is not available when I try to produce to it.

 If I run bin/kafka-console-consumer.sh --topic $topic --zookeeper $zkStr
 --from-beginning it would throw UnknownTopicOrPartitionException.

  Also, I found the following error in controller log.

 [2015-01-23 12:44:41,825] WARN [Controller-0-to-broker-0-send-thread],
 Controller 0 fails to send a request to broker id:0,host:node10,port:9092
 (kafka.controller.RequestSendThread)
 java.io.EOFException: Received -1 when reading from channel, socket has
 likely been closed.
 at kafka.utils.Utils$.read(Utils.scala:381)
 at

 kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
 at
 kafka.network.Receive$class.readCompletely(Transmission.scala:56)
 at

 kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
 at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
 at

 kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
 [2015-01-23 12:44:41,827] ERROR [Controller-0-to-broker-0-send-thread],
 Controller 0 epoch 3 failed to send request

 Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:3;CorrelationId:80;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:node10,port:9092;PartitionState:[app1_test_0_testB,0]
 -

 (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:3),ReplicationFactor:1),AllReplicas:0)
 to broker id:0,host:node10,port:9092. Reconnecting to broker.
 (kafka.controller.RequestSendThread)
 java.nio.channels.ClosedChannelException
 at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
 at

 kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
 at

 kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

 I've tried fetching metadata after creating topic to ensure its
 availability but the fetch failed after several round of retries.

 My kafka is built from 0.8.2 trunk running on local node.

 Any thoughts would be appreciated

 Thanks,
 Manu

Re: kafka topic not exist after creation with api

2015-01-23 Thread Manu Zhang 
Thanks Jun.

My build has not included that fix. I'll try out the latest 0.8.2


On Sat Jan 24 2015 at 上午1:47:40 Jun Rao j...@confluent.io wrote:

 Are you on the latest 0.8.2 branch? We did fix KAFKA-1738 a couple of
 months ago which could prevent new topics from created.

 Thanks,

 Jun

 On Thu, Jan 22, 2015 at 9:44 PM, Manu Zhang owenzhang1...@gmail.com
 wrote:

  Hi all ,
  My application creates kafka topic at runtime with AdminUtils.createTopic
  but that topic is not available when I try to produce to it.
 
  If I run bin/kafka-console-consumer.sh --topic $topic --zookeeper $zkStr
  --from-beginning it would throw UnknownTopicOrPartitionException.
 
   Also, I found the following error in controller log.
 
  [2015-01-23 12:44:41,825] WARN [Controller-0-to-broker-0-send-thread],
  Controller 0 fails to send a request to broker id:0,host:node10,port:9092
  (kafka.controller.RequestSendThread)
  java.io.EOFException: Received -1 when reading from channel, socket has
  likely been closed.
  at kafka.utils.Utils$.read(Utils.scala:381)
  at
 
  kafka.network.BoundedByteBufferReceive.readFrom(
 BoundedByteBufferReceive.scala:54)
  at
  kafka.network.Receive$class.readCompletely(Transmission.scala:56)
  at
 
  kafka.network.BoundedByteBufferReceive.readCompletely(
 BoundedByteBufferReceive.scala:29)
  at kafka.network.BlockingChannel.receive(BlockingChannel.scala:
 108)
  at
 
  kafka.controller.RequestSendThread.doWork(ControllerChannelManager.
 scala:146)
  at kafka.utils.ShutdownableThread.run(
 ShutdownableThread.scala:60)
  [2015-01-23 12:44:41,827] ERROR [Controller-0-to-broker-0-send-thread],
  Controller 0 epoch 3 failed to send request
 
  Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:3;
 CorrelationId:80;ClientId:id_0-host_null-port_9092;
 AliveBrokers:id:0,host:node10,port:9092;PartitionState:[
 app1_test_0_testB,0]
  -
 
  (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:3),
 ReplicationFactor:1),AllReplicas:0)
  to broker id:0,host:node10,port:9092. Reconnecting to broker.
  (kafka.controller.RequestSendThread)
  java.nio.channels.ClosedChannelException
  at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
  at
 
  kafka.controller.RequestSendThread.liftedTree1$
 1(ControllerChannelManager.scala:132)
  at
 
  kafka.controller.RequestSendThread.doWork(ControllerChannelManager.
 scala:131)
  at kafka.utils.ShutdownableThread.run(
 ShutdownableThread.scala:60)
 
  I've tried fetching metadata after creating topic to ensure its
  availability but the fetch failed after several round of retries.
 
  My kafka is built from 0.8.2 trunk running on local node.
 
  Any thoughts would be appreciated
 
  Thanks,
  Manu

Re: kafka topic not exist after creation with api

2015-01-23 Thread Harsha 
Manu,
  Are you passing ZkStringSerializer to zkClient .
ZkClient zkClient = new ZkClient(ZK_CONN_STRING,3, 3, new
ZkStringSerializer());
AdminUtils.createTopic(zkClient, topic, 1, 1, props);
-Harsha

On Thu, Jan 22, 2015, at 09:44 PM, Manu Zhang wrote:
 Hi all ,
 My application creates kafka topic at runtime with AdminUtils.createTopic
 but that topic is not available when I try to produce to it.
 
 If I run bin/kafka-console-consumer.sh --topic $topic --zookeeper $zkStr
 --from-beginning it would throw UnknownTopicOrPartitionException.
 
  Also, I found the following error in controller log.
 
 [2015-01-23 12:44:41,825] WARN [Controller-0-to-broker-0-send-thread],
 Controller 0 fails to send a request to broker id:0,host:node10,port:9092
 (kafka.controller.RequestSendThread)
 java.io.EOFException: Received -1 when reading from channel, socket has
 likely been closed.
 at kafka.utils.Utils$.read(Utils.scala:381)
 at
 kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
 at
 kafka.network.Receive$class.readCompletely(Transmission.scala:56)
 at
 kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
 at
 kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
 at
 kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
 at
 kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
 [2015-01-23 12:44:41,827] ERROR [Controller-0-to-broker-0-send-thread],
 Controller 0 epoch 3 failed to send request
 Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:3;CorrelationId:80;ClientId:id_0-host_null-port_9092;AliveBrokers:id:0,host:node10,port:9092;PartitionState:[app1_test_0_testB,0]
 -
 (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:3),ReplicationFactor:1),AllReplicas:0)
 to broker id:0,host:node10,port:9092. Reconnecting to broker.
 (kafka.controller.RequestSendThread)
 java.nio.channels.ClosedChannelException
 at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
 at
 kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
 at
 kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
 at
 kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
 
 I've tried fetching metadata after creating topic to ensure its
 availability but the fetch failed after several round of retries.
 
 My kafka is built from 0.8.2 trunk running on local node.
 
 Any thoughts would be appreciated
 
 Thanks,
 Manu