[
https://issues.apache.org/jira/browse/KAFKA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352010#comment-17352010
]
Britt Bolen edited comment on KAFKA-9531 at 5/26/21, 6:28 PM:
--------------------------------------------------------------
I see a similar issue using both the cli producer and my application. The
producer goes into a loop attempting to look up a hostname that does not exist,
and never stops trying even if the kafka cluster configuration is changed.
I can produce this by setting up a simple 1 node cluster but not putting any
hostnames in dns. I use the IP for bootstrap and then my producer gets a list
of nodes to connect to which are hostnames that aren't in dns...
{code:java}
[2021-05-26 14:27:38,651] WARN [Producer clientId=console-producer] Error
connecting to node vm-cit-vc-kafka-01:9092 (id: 0 rack: null)
(org.apache.kafka.clients.NetworkClient)[2021-05-26 14:27:38,651] WARN
[Producer clientId=console-producer] Error connecting to node
vm-cit-vc-kafka-01:9092 (id: 0 rack: null)
(org.apache.kafka.clients.NetworkClient)java.net.UnknownHostException:
vm-cit-vc-kafka-01 at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193) at
java.net.InetAddress.getAllByName(InetAddress.java:1127) at
org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111) at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:512)
at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:466)
at
org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
at
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1158)
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1046)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:559) at
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242) at
java.lang.Thread.run(Thread.java:748)
{code}
Adding an `advertised.listeners=PLAINTEXT://10.140.100.230:9092` to my
server.properties fixes everything once I restart my producer processes.
I've observed this on 2.1.1, 2.2.2 and 2.8.0.
at no point does it appear that the producer ever attempts to get a new set of
nodes. (I didn't dig into the code to see if it could/would/might)
was (Author: bolen):
I see a similar issue using both the cli producer and my application. The
producer goes into a loop attempting to look up a hostname that does not exist,
and never stops trying even if the kafka cluster configuration is changed.
I can produce this by setting up a simple 1 node cluster but not putting any
hostnames in dns. I use the IP for bootstrap and then my producer gets a list
of nodes to connect to which are hostnames that aren't in dns...
{code:java}
[2021-05-26 14:27:38,651] WARN [Producer clientId=console-producer] Error
connecting to node vm-cit-vc-kafka-01:9092 (id: 0 rack: null)
(org.apache.kafka.clients.NetworkClient)[2021-05-26 14:27:38,651] WARN
[Producer clientId=console-producer] Error connecting to node
vm-cit-vc-kafka-01:9092 (id: 0 rack: null)
(org.apache.kafka.clients.NetworkClient)java.net.UnknownHostException:
vm-cit-vc-kafka-01 at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193) at
java.net.InetAddress.getAllByName(InetAddress.java:1127) at
org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111) at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:512)
at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:466)
at
org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
at
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1158)
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1046)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:559) at
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242) at
java.lang.Thread.run(Thread.java:748)
{code}
{noformat}
{noformat}
Adding an `advertised.listeners=PLAINTEXT://10.140.100.230:9092` to my
server.properties fixes everything once I restart my producer processes.
I've observed this on 2.1.1, 2.2.2 and 2.8.0.
at no point does it appear that the producer ever attempts to get a new set of
nodes. (I didn't dig into the code to see if it could/would/might)
> java.net.UnknownHostException loop on VM rolling update using CNAME
> -------------------------------------------------------------------
>
> Key: KAFKA-9531
> URL: https://issues.apache.org/jira/browse/KAFKA-9531
> Project: Kafka
> Issue Type: Bug
> Components: clients, controller, network, producer
> Affects Versions: 2.4.0
> Reporter: Rui Abreu
> Priority: Major
>
> Hello,
>
> My cluster setup in based on VMs behind DNS CNAME .
> Example: node.internal is a CNAME to either nodeA.internal or nodeB.internal
> Since kafka-client 1.2.1, it has been observed that sometimes Kafka clients
> get stuck on a loop with the exception:
> Example after nodeB.internal is replaced with nodeA.internal
>
> {code:java}
> 2020-02-10T12:11:28.181Z o.a.k.c.NetworkClient [WARN] - [Consumer
> clientId=consumer-6, groupId=consumer.group] Error connecting to node
> nodeB.internal:9092 (id: 2 rack: null)
> java.net.UnknownHostException: nodeB.internal:9092
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
> ~[?:1.8.0_222]
> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
> ~[?:1.8.0_222]
> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
> ~[?:1.8.0_222]
> at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
> ~[stormjar.jar:?]
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:366)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1220)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:365)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:294)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10715$fn__10730$fn__10761.invoke(executor.clj:649)
> ~[storm-core-1.1.3.jar:1.1.3]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> ~[storm-core-1.1.3.jar:1.1.3]
> at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> {code}
>
> The time it spends in the loop is arbitrary, but it seems the client
> effectively stops while this is happening.
> This error contrasts with instances where the client is able to recover on
> its own after a few seconds:
> {code:java}
> 2020-02-08T01:15:37.390Z o.a.k.c.c.i.AbstractCoordinator [INFO] - [Consumer
> clientId=consumer-7, groupId=consumer-group] Group coordinator
> nodeA.internal:9092 (id: 2147483645 rack: null) is unavailable or invalid,
> will attempt rediscovery
>
> 2020-02-08T01:15:37.885Z o.a.k.c.c.i.AbstractCoordinator [INFO] - [Consumer
> clientId=consumer-7, groupId=consumer-group] Discovered group coordinator
> nodeB.internal:9092 (id: 2147483646 rack: null)
> 2020-02-08T01:15:37.885Z o.a.k.c.ClusterConnectionStates [INFO] - [Consumer
> clientId=consumer-7, groupId=consumer-group] Hostname for node 2147483646
> changed from nodeA.internal to nodeB.internal
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
