[
https://issues.apache.org/jira/browse/KAFKA-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216726#comment-15216726
]
khenaidoo nursimulu commented on KAFKA-2426:
--------------------------------------------
Hello,
I am having the same issue (with both kafka 0.9 and 0.8). I have kafka running
in a docker container (ubuntu) and I advertised the host and port of the
physical machine for remote publishers to send events. I get the exception
below in the controller.log file when the kafka server is attempting to connect
to itself using the advertised ip and port. The ip and port is not reachable
from within the container (the advertised host and port are natd to the docker
IP and port 9092). I tried the above workarounds as well as try to change the
iptables rules within the container but nothing seems to work. is there a
config that would get the controller to use localhost instead of the advertised
ip and port?
Thanks
Khen
[2016-03-29 19:09:39,813] WARN [Controller-0-to-broker-0-send-thread],
Controller 0's connection to broker Node(0, 10.1.89.26, 8198) was unsuccessful
(kafka.controller.RequestSendThread)
java.io.IOException: Connection to Node(0, 10.1.89.26, 8198) failed
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$1.apply(NetworkClientBlockingOps.scala:62)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$1.apply(NetworkClientBlockingOps.scala:58)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$kafka$utils$NetworkClientBlockingOps$$pollUntil$extension$2.apply(NetworkClientBlockingOps.scala:106)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$kafka$utils$NetworkClientBlockingOps$$pollUntil$extension$2.apply(NetworkClientBlockingOps.scala:105)
at
kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:129)
at
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
at
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:105)
at
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:58)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:225)
at
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:172)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> A Kafka node tries to connect to itself through its advertised hostname
> -----------------------------------------------------------------------
>
> Key: KAFKA-2426
> URL: https://issues.apache.org/jira/browse/KAFKA-2426
> Project: Kafka
> Issue Type: Bug
> Components: network
> Affects Versions: 0.8.2.1
> Environment: Docker https://github.com/wurstmeister/kafka-docker,
> managed by a Kubernetes cluster, with an "iptables proxy".
> Reporter: Mikaël Cluseau
> Assignee: Jun Rao
>
> Hi,
> when used behind a firewall, Apache Kafka nodes are trying to connect to
> themselves using their advertised hostnames. This means that if you have a
> service IP managed by the docker's host using *only* iptables DNAT rules, the
> node's connection to "itself" times out.
> This is the case in any setup where a host will DNAT the service IP to the
> instance's IP, and send the packet back on the same interface other a Linux
> Bridge port not configured in "hairpin" mode. It's because of this:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/bridge/br_forward.c#n30
> The specific part of the kubernetes issue is here:
> https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-123925060 .
> The timeout involves that the even if partition's leader is elected, it then
> fails to accept writes from the other members, causing a write lock. and
> generating very heavy logs (as fast as Kafka usualy is, but through log4j
> this time ;)).
> This also means that the normal docker case work by going through the
> userspace-proxy, which necessarily impacts the performance.
> The workaround for us was to add a "127.0.0.2 advertised-hostname" to
> /etc/hosts in the container startup script.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
