QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

[Mike Bridge]

I am able to get a simple one-node Kafka (kafka_2.11-0.8.2.1) working
locally on one linux machine, but when I try to run a producer remotely I'm
getting some confusing errors.

I'm following the quickstart guide at
http://kafka.apache.org/documentation.html#quickstart.  I stopped the kafka
processes and deleted all the zookeeper & karma files in /tmp.  I am on a
local 10.0.0.0/24 network NAT-ed with an external IP address, so I modified
`server.properties` to tell zookeeper how to broadcast my external address:

    advertised.host.name=MY.EXTERNAL.IP

Then I'm running this:

    $ bin/zookeeper-server-start.sh config/zookeeper.properties
    --> ...
    $ export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # small test server!
    $ bin/kafka-server-start.sh config/server.properties
    --> ...

I opened up the firewall for my producer on the remote machine, and created
a new topic and verified it:

    $ bin/kafka-topics.sh --create --zookeeper MY.EXTERNAL.IP:2181
--replication-factor 1 --partitions 1 --topic test123
    --> Created topic "test123".
    $ bin/kafka-topics.sh --list --zookeeper MY.EXTERNAL.IP:2181
    --> test123

However, the producer I'm running remotely gives me errors:

    $ bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092
--topic test123
    --> [2015-06-16 14:41:19,757] WARN Property topic is not valid
(kafka.utils.VerifiableProperties)
    My Test Message
    --> [2015-06-16 14:42:43,347] WARN Error while fetching metadata
[{TopicMetadata for topic test123 ->
No partition metadata for topic test123 due to
kafka.common.LeaderNotAvailableException}] for topic [test123]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
    --> (repeated several times)

(I disabled the whole firewall to make sure that wasn't the problem.)

The stdout errors in the karma-startup are repeated: `[2015-06-16
20:42:42,768] INFO Closing socket connection to /MY.EXTERNAL.IP.
(kafka.network.Processor)`

And the controller.log gives me this, several times:

    java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
        at
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
        at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
    [2015-06-16 20:44:08,128] INFO [Controller-0-to-broker-0-send-thread],
Controller 0 connected to id:0,host:MY.EXTERNAL.IP,port:9092 for sending
state change requests (kafka.controller.RequestSendThread)
    [2015-06-16 20:44:08,428] WARN [Controller-0-to-broker-0-send-thread],
Controller 0 epoch 1 fails to send request
Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:7;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:MY.EXTERNAL.IP,port:9092;PartitionState:(test123,0)
->
(LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)
to broker id:0,host:MY.EXTERNAL.IP,port:9092. Reconnecting to broker.
(kafka.controller.RequestSendThread)


Any idea what might be wrong?

-Mike