zey1996 opened a new issue, #23843: URL: https://github.com/apache/pulsar/issues/23843
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version k8s with ipv6 imag version:3.3.3 no client, just cli. ### Minimal reproduce step this is my broker deploy yaml file: ```yaml --- apiVersion: apps/v1 kind: StatefulSet metadata: name: "my-release-pulsar-broker" namespace: milvus labels: app: pulsar chart: pulsar-2.7.8 release: my-release heritage: Helm cluster: my-release-pulsar component: broker spec: serviceName: "my-release-pulsar-broker" replicas: 3 selector: matchLabels: app: pulsar release: my-release component: broker updateStrategy: type: RollingUpdate podManagementPolicy: Parallel template: metadata: labels: app: pulsar release: my-release cluster: my-release-pulsar component: broker annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: broker component: bookie topologyKey: kubernetes.io/hostname weight: 1 serviceAccountName: "my-release-pulsar-broker-acct" terminationGracePeriodSeconds: 30 initContainers: - name: wait-zookeeper-ready image: "registry.paas/eos-indexer/pulsar:3.3.3" imagePullPolicy: IfNotPresent command: ["sh", "-c"] args: - >- until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server my-release-pulsar-zookeeper:2181 get /admin/clusters/my-release-pulsar; do echo "pulsar cluster my-release-pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done; volumeMounts: - name: wait-bookkeeper-ready image: "registry.paas/eos-indexer/pulsar:3.3.3" imagePullPolicy: IfNotPresent command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/bookkeeper.conf; until bin/bookkeeper shell whatisinstanceid; do echo "bookkeeper cluster is not initialized yet. backoff for 3 seconds ..."; sleep 3; done; echo "bookkeeper cluster is already initialized"; bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)"; until [ ${bookieServiceNumber} -ge 2 ]; do echo "bookkeeper cluster my-release-pulsar isn't ready yet ... check in 10 seconds ..."; sleep 10; bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)"; done; echo "bookkeeper cluster is ready"; envFrom: - configMapRef: name: "my-release-pulsar-bookie" volumeMounts: containers: - name: "my-release-pulsar-broker" image: "registry.paas/eos-indexer/pulsar:3.3.3" imagePullPolicy: IfNotPresent resources: requests: cpu: 1.5 memory: 4096Mi command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/broker.conf; bin/gen-yml-from-env.py conf/functions_worker.yml; echo "OK" > status; bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080; while [ $? -eq 0 ]; do echo "broker ${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local znode still exists ... check in 10 seconds ..."; sleep 10; bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080; done; cat conf/pulsar_env.sh; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar broker; ports: - name: http containerPort: 8080 hostPort: 8080 - name: "pulsar" containerPort: 6650 hostPort: 6650 envFrom: - configMapRef: name: "my-release-pulsar-broker" volumeMounts: volumes: --- apiVersion: v1 kind: ConfigMap metadata: name: "my-release-pulsar-broker" namespace: milvus labels: app: pulsar chart: pulsar-2.7.8 release: my-release heritage: Helm cluster: my-release-pulsar component: broker data: zookeeperServers: "my-release-pulsar-zookeeper:2181" configurationStoreServers: "my-release-pulsar-zookeeper:2181" clusterName: my-release-pulsar exposeTopicLevelMetricsInPrometheus: "true" numHttpServerThreads: "8" zooKeeperSessionTimeoutMillis: "30000" statusFilePath: "/pulsar/status" functionsWorkerEnabled: "false" webServicePort: "8080" brokerServicePort: "6650" PULSAR_GC: | -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError PULSAR_MEM: | -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m backlogQuotaDefaultLimitGB: "32" backlogQuotaDefaultRetentionPolicy: producer_exception defaultRetentionSizeInMB: "-1" defaultRetentionTimeInMinutes: "10080" managedLedgerDefaultAckQuorum: "1" managedLedgerDefaultEnsembleSize: "1" managedLedgerDefaultWriteQuorum: "1" maxMessageSize: "104857600" subscriptionExpirationTimeMinutes: "3" ttlDurationDefaultInSeconds: "259200" --- apiVersion: v1 kind: Service metadata: name: "my-release-pulsar-broker" namespace: milvus labels: app: pulsar chart: pulsar-2.7.8 release: my-release heritage: Helm cluster: my-release-pulsar component: broker spec: ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: http port: 8080 - name: "pulsar" port: 6650 clusterIP: None selector: app: pulsar release: my-release component: broker ``` ### What did you expect to see? I am a newer for pulsar, I just want to run a demo of pulsar cluster with some node. first. I deploy my pods in k8s cluster. they are look good. then, I want to create a topic, but I'm failed. ### What did you see instead? in pulsar-admin cli, I got this. ``` bash my-release-pulsar-proxy-0:/pulsar# pulsar-admin topics create public/default/test1 2025-01-13T07:41:27,924+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/default/test1] Failed to perform http put request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout HTTP 504 Gateway Timeout Reason: HTTP 504 Gateway Timeout ``` then I found some exception log in broker pod: ``` 2025-01-13T07:41:04,652+0000 [pulsar-io-3-12] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 6.085 s 2025-01-13T07:41:10,739+0000 [broker-client-shared-timer-8-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Reconnecting after connection was closed 2025-01-13T07:41:10,742+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local/<unresolved>:6650 : io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] 2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://public/default/__change_events] lookup failed : java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] at org.apache.pulsar.client.impl.ConnectionPool.lambda$createConnection$13(ConnectionPool.java:333) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:408) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at java.base/java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$18(ConnectionPool.java:409) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86) ~[io.netty-netty-resolver-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext$1.operationComplete(DnsResolveContext.java:268) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1158) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] ... 4 more Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1150) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final] ... 4 more Caused by: io.netty.resolver.dns.DnsNameResolverException: [4836: /[fd11:1111:1112:15:0:0:0:a]:53] DefaultDnsQuestion(my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local. IN A) failed to send a query '4836' via UDP (no stack trace available) Caused by: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol 2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] 2025-01-13T07:41:10,744+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 12.786 s ``` It appears that pulsar cannot use ipv6 dns server? I confirm the dns server in log is working. ``` bash my-release-pulsar-broker-0:/pulsar$ nslookup my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local fd11:1111:1112:15:0:0:0:a Server: fd11:1111:1112:15:0:0:0:a Address: fd11:1111:1112:15::a#53 Name: my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local Address: 10.222.55.247 ``` Can I run pulsar in ipv6 k8s cluster? ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
