[
https://issues.apache.org/jira/browse/SAMZA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shanthoosh Venkataraman updated SAMZA-1586:
-------------------------------------------
Description:
Embedded kafka broker was unreachable in standalone integration tests and kafka
NetworkClient spams the logs with the following exception messages.
{code:java}
99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to
java.nio.channels.ClosedChannelException. Releasing ownership of all
partitions, and restarting consumer. Turn on debugging to get a full stack
trace.
99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
[test_ZKS_numbers1,0]
99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
[test_ZKS_numbers1,1]
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
Map([test_ZKS_numbers1,1] -> 20)
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
Map([test_ZKS_numbers1,0] -> 20)
99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
for client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
for client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
{code}
Unavailability of kafka broker causes the SystemConsumer.poll() and
SystemProducer.flush() to fail with exceptions and the tests fail with junit
timeout exceeded error.
Actual message processing is stalled by the infinite retry loop in BrokerProxy
I/O thread which polls messages from the embedded kafka broker.
This requires further investigation.
was:
Embedded kafka broker was unreachable in standalone integration tests and kafka
NetworkClient spams the logs with the following exception messages.
{code:java}
99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to
java.nio.channels.ClosedChannelException. Releasing ownership of all
partitions, and restarting consumer. Turn on debugging to get a full stack
trace.
99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
[test_ZKS_numbers1,0]
99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
[test_ZKS_numbers1,1]
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
Map([test_ZKS_numbers1,1] -> 20)
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] INFO
org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
Map([test_ZKS_numbers1,0] -> 20)
99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for
client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
for client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
for client samza_consumer-test_job-1] WARN
org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata
for topics [Set(test_ZKS_numbers1)] from broker
[ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
established. Broker may not be available.
{code}
This causes the standalone integrations tests to fail with junit timeout
exceeded error.
Message processing by stream processors is stalled by the infinite retry loop
in BrokerProxy I/O thread which polls messages from the embedded kafka broker.
This requires further investigation.
> Kafka broker setup problem in standalone integration tests.
> -----------------------------------------------------------
>
> Key: SAMZA-1586
> URL: https://issues.apache.org/jira/browse/SAMZA-1586
> Project: Samza
> Issue Type: Bug
> Reporter: Shanthoosh Venkataraman
> Assignee: Shanthoosh Venkataraman
> Priority: Minor
>
> Embedded kafka broker was unreachable in standalone integration tests and
> kafka NetworkClient spams the logs with the following exception messages.
> {code:java}
> 99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to
> java.nio.channels.ClosedChannelException. Releasing ownership of all
> partitions, and restarting consumer. Turn on debugging to get a full stack
> trace.
> 99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] INFO
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [test_ZKS_numbers1,0]
> 99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] INFO
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [test_ZKS_numbers1,1]
> 99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] INFO
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([test_ZKS_numbers1,1] -> 20)
> 99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] INFO
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([test_ZKS_numbers1,0] -> 20)
> 99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243
> for client samza_consumer-test_job-1] WARN
> org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers
> for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic
> metadata for topics [Set(test_ZKS_numbers1)] from broker
> [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> 100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN
> org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be
> established. Broker may not be available.
> {code}
> Unavailability of kafka broker causes the SystemConsumer.poll() and
> SystemProducer.flush() to fail with exceptions and the tests fail with junit
> timeout exceeded error.
> Actual message processing is stalled by the infinite retry loop in
> BrokerProxy I/O thread which polls messages from the embedded kafka broker.
> This requires further investigation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
