Sounds like you need to use advertised.host configuration with the external name / ip.
This means that the broker will send producers / consumers / zookeeper their external address and they will be able to connect. Gwen On Tue, Dec 8, 2015 at 11:17 AM, Henrik Martin <hen...@netgate.net> wrote: > Greetings. Apologies for the verbose email, but I'm trying to provide as > much relevant detail as possible. I have an Amazon AWS server that is > running 1 instance of Zookeeper, and 1 instance of Kafka 0.9.0. As all the > AWS servers, it has an internal non routable IP address (172.X.X.X), and an > external NATed IP address (54.X.X.X). Zookeeper is binding to the default > interface. I'm using the Java system property java.net.preferIPv4Stack=true > so that Zookeeper and Kafka bind to the IPv4 interface. When I run netstat, > it looks like this: > > tcp 0 0 0.0.0.0:9092 0.0.0.0:* LISTEN > tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN > > I can successfully run the Kafka consumer/producer scripts from the same > AWS machine, against either localhost, 127.0.0.1, or the internal IP > address (partially masked here): > > echo "test" | bin/kafka-console-producer.sh --broker-list localhost:9092 > --topic test (works fine) > echo "test2" | bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 > --topic test (works fine) > echo "test3" | bin/kafka-console-producer.sh --broker-list 172.X.X.X:9092 > --topic test (IP address obfuscated here, but works fine). > > I can read the messages: > > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test > --from-beginning > test > test2 > test3 > > I have also configured Kafka to use the external host name in the server > config properties file (host name obfuscated intentionally for this email): > > advertised.host.name=ec2-54-....compute.amazonaws.com > > So far, so good. It's when I try to access Kafka remotely that I'm running > into problems. I have granted access to connect to all TCP ports on the AWS > machine from my VPN server. From a remote machine on the VPN, I can connect > to the ports for Zookeeper and Kafka. However, I can't seem to access the > queue. I've tried both the consumer and producer scripts, as well as the > "kafkacat" program. I get slightly different error messages. This is what I > see when using kafkacat running on my laptop over the VPN against the > external IP address of the AWS Zookeeper/Kafka machine: > > kafkacat -b 54.X.X.X:9092 -o beginning -t test > % Auto-selecting Consumer mode (use -P or -C to override) > %3|1449600975.259|FAIL|rdkafka#consumer-0| ec2-54-.... > us-west-1.compute.amazonaws.com:9092/0: Failed to connect to broker at > ip-172-....us-west-1.compute.internal:9092: Operation timed out > %3|1449600975.259|ERROR|rdkafka#consumer-0| ec2-54-.... > us-west-1.compute.amazonaws.com:9092/0: Failed to connect to broker at > ip-172-....us-west-1.compute.internal:9092: Operation timed out > > The timeout happens after several minutes. What I find interesting is that > it prints the broker IP using the internal hostname for the AWS machine. I > guess that might make sense if the error message is coming from the remote > ZK instance? I can list the topics from the laptop on the VPN: > > kafka-topics.sh --zookeeper 54.X.X.X:2181 --list > test > > When I'm running the producer script I get this: > > echo hello | bin/kafka-console-producer.sh --topic test --broker-list > 54.X.X.X:9092 > [2015-12-08 11:09:12,362] ERROR Error when sending message to topic test > with key: null, value: 5 bytes with error: Failed to update metadata after > 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > And here's with the consumer, trying to consume messages that are already > on the queue. Same thing, consumer running on laptop over the VPN: > > bin/kafka-console-consumer.sh --topic test --zookeeper 54.X.X.X:2181 > --from-beginning > [2015-12-08 11:13:21,205] WARN Fetching topic metadata with correlation id > 0 for topics [Set(test)] from broker [BrokerEndPoint(0,ec2-54-.... > us-west-1.compute.amazonaws.com,9092)] failed (kafka.client.ClientUtils$) > java.nio.channels.ClosedChannelException > at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) > at > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74) > at kafka.producer.SyncProducer.send(SyncProducer.scala:119) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) > at > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > [2015-12-08 11:13:21,209] WARN > [console-consumer-38323_ip-192-168-4-28.us-west-1.compute.internal-1449601968243-29980a1b-leader-finder-thread], > Failed to find leader for Set([test,0]) > (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread) > kafka.common.KafkaException: fetching topic metadata for topics > [Set(test)] from broker [ArrayBuffer(BrokerEndPoint(0,ec2-54-.... > us-west-1.compute.amazonaws.com,9092))] failed > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) > at > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > Caused by: java.nio.channels.ClosedChannelException > at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) > at > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74) > at kafka.producer.SyncProducer.send(SyncProducer.scala:119) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) > ... 3 more > > The versions I'm running are: > > Zookeeper: 3.4.5--1, built on 06/10/2013 17:26 GMT > Kafka: kafka_2.11-0.9.0.0 > Java: Oracle JDK, version 1.8.0_66 > > I'm sure it's just a configuration issue. Any help resolving this is > greatly appreciated. Thanks, > > /Henrik > > > > >
