frankjkelly opened a new issue, #21206: URL: https://github.com/apache/pulsar/issues/21206
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version OS: Linux Pulsar: 2.10.3 Chart: 3.0.0 ### Minimal reproduce step 1) Autoscale Brokers by driving load through pulsar 2) Have a Pulsar client connection and operation time around 3 to 4 seconds ### What did you expect to see? Proxies quickly detect any new brokers within a few seconds ### What did you see instead? Proxy DNS Lookup failures result in client failures ``` 2023-09-15T14:25:06,331+0000 [pulsar-proxy-io-2-1] ERROR org.apache.pulsar.proxy.server.ProxyConnection - [/127.0.0.6:50675] Error validating target broker 'platform-pulsar-broker-4.platform-pulsar-broker.cogito-load.svc.cluster.local:6650'. authenticated with none role null. java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'platform-pulsar-broker-4.platform-pulsar-broker.cogito-load.svc.cluster.local' after 2 queries at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?] at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.apache.pulsar.common.util.netty.NettyFutureUtil.lambda$toCompletableFuture$0(NettyFutureUtil.java:51) ~[org.apache.pulsar-pulsar-common-2.10.3.jar:2.10.3] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.InetSocketAddressResolver$1.operationComplete(InetSocketAddressResolver.java:62) ~[io.netty-netty-resolver-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: java.net.UnknownHostException: Failed to resolve 'platform-pulsar-broker-4.platform-pulsar-broker.cogito-load.svc.cluster.local' after 2 queries at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:971) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:66) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:471) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:232) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:224) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314) ~[io.netty-netty-resolver-dns-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollDatagramChannel.processPacket(EpollDatagramChannel.java:662) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollDatagramChannel.recvmsg(EpollDatagramChannel.java:697) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollDatagramChannel.access$300(EpollDatagramChannel.java:56) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:536) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] ... 4 more", ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
