zzzming opened a new issue, #814: URL: https://github.com/apache/pulsar-client-go/issues/814
In a Pulsar cluster's kubernetes deployment or a deployment with Proxy/LB in the front, we need metrics counter to track the re-connection failure producers and consumers. When brokers go offline but the proxy/LB is still functioning, TCP connection can still be established but the topic look up failed. `pulsar_client_connections_establishment_errors` counter is not incremented in this case. We have clients built-in failover to other clusters based on counter. Therefore new counters are required to track such failure cases. This case is to track reconnect failure. When the brokers are offline upon initial connection, the establishment error counter is used. It is not covered by this issue. Here is the complete client log from the test. ```$ go run examples/producer/producer.go INFO[0000] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0000] [TCP connection established] local_addr="192.168.1.124:57557" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0000] [Connection is ready] local_addr="192.168.1.124:57557" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0000] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0001] [TCP connection established] local_addr="192.168.1.124:57558" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0001] [Connection is ready] local_addr="192.168.1.124:57558" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0001] [Connected producer] cnx="192.168.1.124:57558 -> 34.236.188.146:6651" epoch=0 topic="persistent://mingawsuseast1/default/topic2" INFO[0001] [Created producer] cnx="192.168.1.124:57558 -> 34.236.188.146:6651" producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ... sending messages ... and stop all the brokers to simulate broker side failure ... the proxy/LB are still functioning INFO[0034] Broker notification of Closed producer: 1 local_addr="192.168.1.124:57558" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0034] [Connection was closed] cnx="192.168.1.124:57558 -> 34.236.188.146:6651" producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0034] [runEventsLoop will reconnect in producer] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0034] [Reconnecting to broker in 119.204716ms] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" WARN[0036] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0036] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0036] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0036] [Reconnecting to broker in 216.849167ms] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0036] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0036] [TCP connection established] local_addr="192.168.1.124:57578" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0036] [Connection is ready] local_addr="192.168.1.124:57578" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0037] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0037] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0037] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0037] [Reconnecting to broker in 407.84384ms] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0038] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0038] [TCP connection established] local_addr="192.168.1.124:57583" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0038] [Connection is ready] local_addr="192.168.1.124:57583" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0039] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0039] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0039] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0039] [Reconnecting to broker in 818.646893ms] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0040] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0040] [TCP connection established] local_addr="192.168.1.124:57585" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0040] [Connection is ready] local_addr="192.168.1.124:57585" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0041] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0041] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0041] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0041] [Reconnecting to broker in 1.716085644s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0043] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0043] [TCP connection established] local_addr="192.168.1.124:57586" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0043] [Connection is ready] local_addr="192.168.1.124:57586" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0044] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0044] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0044] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0044] [Reconnecting to broker in 3.614064069s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0048] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0048] [TCP connection established] local_addr="192.168.1.124:57588" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0048] [Connection is ready] local_addr="192.168.1.124:57588" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0049] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0049] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0049] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0049] [Reconnecting to broker in 7.271431284s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0056] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0056] [TCP connection established] local_addr="192.168.1.124:57589" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0057] [Connection is ready] local_addr="192.168.1.124:57589" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0058] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0058] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0058] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0058] [Reconnecting to broker in 13.853772227s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0061] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0061] [TCP connection established] local_addr="192.168.1.124:57591" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0061] [Connection is ready] local_addr="192.168.1.124:57591" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0071] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0072] [TCP connection established] local_addr="192.168.1.124:57592" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0073] [Connection is ready] local_addr="192.168.1.124:57592" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0074] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0074] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0074] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0074] [Reconnecting to broker in 26.208890579s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0100] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0101] [TCP connection established] local_addr="192.168.1.124:57616" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0101] [Connection is ready] local_addr="192.168.1.124:57616" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0102] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0102] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0102] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0102] [Reconnecting to broker in 1m1.126503091s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0121] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0121] [TCP connection established] local_addr="192.168.1.124:57620" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0121] [Connection is ready] local_addr="192.168.1.124:57620" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0163] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0164] [TCP connection established] local_addr="192.168.1.124:57625" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0164] [Connection is ready] local_addr="192.168.1.124:57625" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0165] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0165] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0165] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0165] [Reconnecting to broker in 1m11.194668451s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0181] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0181] [TCP connection established] local_addr="192.168.1.124:57628" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0181] [Connection is ready] local_addr="192.168.1.124:57628" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0236] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0236] [TCP connection established] local_addr="192.168.1.124:57639" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0236] [Connection is ready] local_addr="192.168.1.124:57639" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0237] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0237] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0237] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0237] [Reconnecting to broker in 1m2.537721342s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0241] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0241] [TCP connection established] local_addr="192.168.1.124:57641" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0241] [Connection is ready] local_addr="192.168.1.124:57641" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0300] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0300] [TCP connection established] local_addr="192.168.1.124:57648" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0300] [Connection is ready] local_addr="192.168.1.124:57648" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0301] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0301] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0301] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0301] [Reconnecting to broker in 1m2.284926552s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0361] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0361] [TCP connection established] local_addr="192.168.1.124:57660" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0361] [Connection is ready] local_addr="192.168.1.124:57660" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0364] [Connecting to broker] remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0364] [TCP connection established] local_addr="192.168.1.124:57661" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" INFO[0364] [Connection is ready] local_addr="192.168.1.124:57661" remote_addr="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" WARN[0365] [Failed to lookup topic] error=ServiceNotReady message="org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: pulsar-aws-useast1-staging-broker.pulsar.svc.cluster.local/10.100.188.109:6650" serviceURL="pulsar+ssl://pulsar-aws-useast1.staging.streaming.datastax.com:6651" topic="persistent://mingawsuseast1/default/topic2" WARN[0365] [Failed to lookup topic] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ERRO[0365] [Failed to create producer at reconnect] error=ServiceNotReady producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" INFO[0365] [Reconnecting to broker in 1m11.629810795s] producerID=1 producer_name=pulsar-aws-useast1-staging-137-34 topic="persistent://mingawsuseast1/default/topic2" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
