[ 
https://issues.apache.org/jira/browse/SAMZA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shanthoosh Venkataraman updated SAMZA-1586:
-------------------------------------------
    Description: 
Embedded kafka broker was unreachable in standalone integration tests and kafka 
NetworkClient spams the logs with the following exception messages.
{code:java}
    99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
java.nio.channels.ClosedChannelException. Releasing ownership of all 
partitions, and restarting consumer. Turn on debugging to get a full stack 
trace.

    99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,0]

    99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,1]

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,1] -> 20)

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,0] -> 20)

    99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

 {code}
This causes the standalone integrations tests to fail with junit timeout 
exceeded error. 

Message processing by stream processors is stalled by the infinite retry loop 
in BrokerProxy I/O thread which polls messages from the embedded kafka broker.

This requires further investigation.

  was:
We've observed in standalone integration tests that the embedded kafka broker 
was unreachable and kafka NetworkClient spams the logs with the following 
exception messages.
{code:java}
    99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
java.nio.channels.ClosedChannelException. Releasing ownership of all 
partitions, and restarting consumer. Turn on debugging to get a full stack 
trace.

    99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,0]

    99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
[test_ZKS_numbers1,1]

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,1] -> 20)

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] INFO 
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
Map([test_ZKS_numbers1,0] -> 20)

    99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for 
client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
for client samza_consumer-test_job-1] WARN 
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata 
for topics [Set(test_ZKS_numbers1)] from broker 
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

    100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
established. Broker may not be available.

 {code}
This causes the standalone integrations tests to fail with junit timeout 
exceeded error. 

Message processing by stream processors is stalled by the infinite retry loop 
in BrokerProxy I/O thread which polls messages from the embedded kafka broker.

This requires further investigation.


> Kafka broker setup problem in standalone integration tests.
> -----------------------------------------------------------
>
>                 Key: SAMZA-1586
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1586
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Minor
>
> Embedded kafka broker was unreachable in standalone integration tests and 
> kafka NetworkClient spams the logs with the following exception messages.
> {code:java}
>     99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to 
> java.nio.channels.ClosedChannelException. Releasing ownership of all 
> partitions, and restarting consumer. Turn on debugging to get a full stack 
> trace.
>     99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
> [test_ZKS_numbers1,0]
>     99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for 
> [test_ZKS_numbers1,1]
>     99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
> Map([test_ZKS_numbers1,1] -> 20)
>     99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] INFO 
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: 
> Map([test_ZKS_numbers1,0] -> 20)
>     99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 
> for client samza_consumer-test_job-1] WARN 
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers 
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(test_ZKS_numbers1)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
>     100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>     100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN 
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be 
> established. Broker may not be available.
>  {code}
> This causes the standalone integrations tests to fail with junit timeout 
> exceeded error. 
> Message processing by stream processors is stalled by the infinite retry loop 
> in BrokerProxy I/O thread which polls messages from the embedded kafka broker.
> This requires further investigation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to