[ 
https://issues.apache.org/jira/browse/SAMZA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shanthoosh Venkataraman updated SAMZA-1586:
-------------------------------------------
    Description: 
We've observed in standalone integration tests that the embedded kafka broker 
was unreachable and kafka NetworkClient spams the logs with the following 
exception messages.
{code:java}
    99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
java.nio.channels.ClosedChannelException. Releasing ownership of all 
partitions, and restarting consumer. Turn on debugging to get a full stack 
trace.

    99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,0]

    99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,1]

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,1] -> 20)

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,0] -> 20)

    99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

 {code}
This causes the standalone integrations tests to fail with junit timeout 
exceeded error. 

Message processing by stream processors is stalled by the infinite retry loop 
in BrokerProxy I/O thread which polls messages from the embedded kafka broker.

This requires further investigation.

  was:
We've observed in standalone integration tests that embedded kafka broker was 
unreachable and kafka NetworkClient spams the logs with the following exception 
messages.
{code:java}
    99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
java.nio.channels.ClosedChannelException. Releasing ownership of all 
partitions, and restarting consumer. Turn on debugging to get a full stack 
trace.

    99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,0]

    99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,1]

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,1] -> 20)

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,0] -> 20)

    99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

{code}
 

Reading/writing into kafka within the integration test fails because of the 
above problem.

Majority of the standalone integration tests that fail with junit test timeout 
were stuck in the infinite retry loop of BrokerProxy I/O thread which polls 
messages from the embedded kafka broker.

This requires further investigation.


> Kafka broker setup problem in standalone integration tests.
> -----------------------------------------------------------
>
>                 Key: SAMZA-1586
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1586
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Minor
>
> We've observed in standalone integration tests that the embedded kafka broker 
> was unreachable and kafka NetworkClient spams the logs with the following 
> exception messages.
> {code:java}
>     99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
> java.nio.channels.ClosedChannelException. Releasing ownership of all 
> partitions, and restarting consumer. Turn on debugging to get a full stack 
> trace.
>     99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
> [test_ZKS_numbers1,0]
>     99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
> [test_ZKS_numbers1,1]
>     99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
> Map([test_ZKS_numbers1,1] -> 20)
>     99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
> Map([test_ZKS_numbers1,0] -> 20)
>     99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>  {code}
> This causes the standalone integrations tests to fail with junit timeout 
> exceeded error. 
> Message processing by stream processors is stalled by the infinite retry loop 
> in BrokerProxy I/O thread which polls messages from the embedded kafka broker.
> This requires further investigation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to