stubbornTanzhe opened a new issue, #9737: URL: https://github.com/apache/apisix/issues/9737
### Description stateful-set etcd cluster with ipv6 address, when apisix pod init, it failed because connect to etcd domain url(headless service) fail. It shows ``` request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known Warning! Request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known, retry time=1 ``` curl the etcd url in apisix pod is success. and I change the etcd url from string to ip-address(in the configmap), it shows ``` /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported request etcd endpoint 'http://[fd00:c5a6::1b:7478]:2379/version' error, Address family for hostname not supported request etcd endpoint 'http://[fd00:c5a6::27:3dfb]:2379/version' error, Address family for hostname not supported all etcd nodes are unavailable Warning! Request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported, retry time=1 ``` and the openresty version is with ipv6: ``` [root@5927f3188b04 resty]# /usr/local/openresty/bin/openresty -p /usr/local/apisix -V nginx version: openresty/1.21.4.1 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1s 1 Nov 2022 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.7 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/ local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/ local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit -0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-m isc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../sr cache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0 .33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-mod ule=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpat h,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl1 11/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp. OdVMn6DSl0/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 - -add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../a pisix-nginx-module-1.12.0/src/stream --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/meta --add-mod ule=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../wasm-nginx-module-0.6.4 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../lua-var-ngin x-module-v0.5.3 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../grpc-client-nginx-module-v0.4.2 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_ imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_a uth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module - -with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-ipv6 -- with-stream --with-http_ssl_module ``` now i am confused 1. init-etcd seems that it just use socket http lib send request(https://github.com/apache/apisix/blob/master/apisix/cli/etcd.lua#L135) 2. i found some issue which with upstream ipv6 and etcd ipv6 ,but close them already(https://github.com/apache/apisix/issues/7100), i do not get the right solution. in my version(2.15.3), it just NOT work. etcd yaml: ``` apiVersion: v1 data: jwt-token.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS2dJQkFBS0NBZ0VBN1d6LzVNb3AzVHF0QWNMeFVMU2p5Mm9SbmhJSXBvTndBV2pLb2Uxb08reDFlKzMzCjc1dG1BTU91SmhxS0c3TGhIdFVxWENiQXZlYWRyZ09QTnJ4MGVLS0J2Ni9sY0FBbUh2Y2FVdyt3d2lYNFFxWXgKSWE0NTVGODkyL3pBT2YwUm5UMVVjUXJQbDR3SDZWWndvY3QrNzhZL3lDaWRQcGpoWTNra0IveWszUmlxUVFMMwpqR3NHMXZLYURNUEl3bFgxc0xkRUdzTjhjN1pvd0x5SWtCc1BiSWlYSHdPeWE3Mkp2cVRZc2dydWw5Ri8yRFl1ClBWZEhaMzREdTBpd1NVNCt1N1lQS0lFYUp6RmdSd3c2czJXYlZBQXVEN1h6LzU2anMvTm5uZEFNWHlYT1RReGsKTnQwZlZVS2R5SWNPdGJGK2w0blFnNG1RVXBDS3MvdGhHYzFzY1hMWHpqWVJaclo3NjFkVFRYSm15UWRWbWhOYgpaemZ2SzJEZ3ppYkZBcStPQ29PaWE2TlByYUhTZEFsejlWcS92MFB4T0RlbXZDR2Q1clg2VHF4eUdqcGtGcWxOCmdNbVpWR2pHdVBEWXN5TVpjUDZOWEVoUFFhWmRWNzY5Q1kyY0xlSWVsWk1LNC9TUDNOL0RiU3E2cEdZaDhOV1kKV3pyV2k0UWtyUVI0UnBlOGhGL1FRbmo5RnpVTUxlR05zUG91L0JnTGJCbDIyN2E3NmdlUEsrMmp2VW5UMXlCVApLbzJmWFlnYTgxVEhpYzhOR2VCSDlrU1VQMThXVUVTcGI2cXVJTVQ5dHAzTmx6SDdYc3E4TnNqQlEwMVJqb2VCCm9Tc1hYcDZteWhJdU90QjZINkFLbTJwTitCU0lrRW9XL0RUYlJwTllkdE5FSjRpT3BWZF NYaFE1d1dzQ0F3RUEKQVFLQ0FnRUE0R3VHV2NGSjN3WkdZQUxmVExUaW1qU3hZR09WYjZlUHFKVmY2NzFYS0VVeks2aGRFa3dtRjVUZgpPTDhmU2wxRDgzdjBaVlJ3SFZBdVBUeTZFemdCTThJUHVKdlAySlViYzJ1SW8zaVNVSUhKYjE1ZDFJTTF6NHJECkV2eU95b0ZPTS8yem9vSmdoTlBrNm9jOVFZSHpnenMyTU1GTEdNOHpSRzVPUTRwb1gyQ2dGWUZ6Y25QNzdCTTIKejlnRUhNVlp6ZWI4NWZ0ZjJKUlQ0ZEtUbHZzdEVoN3VFVy9TSjRycDk1MGtBdTdZWmdLdHdEWStjTmJkOUpJbApmdEtQWjBzaGV4YjByd21SS2ZpL0U0a1l0MWZVZ1lzRi91MnpMQ3pNZWgwYUZRNkJrekpPRllBZi9jZjNoUVRKCjFiRHJrNHlwcHgycVRac3JJdy9qTzRyaVZweFk0ZlVjOW5MeVJPeG1CaC9uTHlFNFVEQWhZSGMwZjZhUWxYSVYKcGtpZ3d4ZENrditRNm9uaUZLUFU0RDF3VlhQOVVxZ2ltaFJ5V2tPcmxXUGZWa09laWIwek03Z3NvVXBtRGJ4TAoyOXAweHB6WVBPQ3REbXVXTEZxZWZmSUFFM0g5UFowWHVRUlJPQTV3cHlBdUVqK1U1cjJPMGQ5T0ZLQVA0akpXCmM3aHEwK29pZHhHNmZMK3VJZ0d1T1hPZW9uYU5jNFQ2NVZPbG9EQWkrdkZycUlXVktSbm9qMjFLdytyYzg0aHoKK2tPTktnbHVjZ2JjbUgrb1AvanU3dklTdW8raXpRUFViZnZ2alhwRThxc1l0eW1UY1J2UW5pd2I2c3h0ekNBVgpuZVE5ZElZMGYzT2JvQitucHBQMUFIZWszZlJLN1hZZ2c5eTJ4eE1IdFpyRitvUXNpaGtDZ2dFQkFPNHNzVnVLCmhLcWNJcmdkZUc2OEg0TGNtT09 GWHRyZlRMNTVzTS83STI5UWhjMmczc3pYYXZoM1BndEo1ekxJeDMwdWdKdWYKN1h6Snh2Tjl2RGs1SUlVTXc3SnRJYXZKNGVQWks2L0lHZUVkOXhWTUF2NTIrVE5rOFJSbW5oeW5ua0FPNlkxcwpGM21GTURmblVxbFhzODRCVlM1N1Y3Z3RZMXV2TWpOMHRSdmdjNkxFeStpd1NQYTRnUXM2aFY4R3dMSVk3ekQvCmJHWU93bVgxNjhwbVBhbXFxOXIyUU1ubnNiWFhJb3R5aVZ1YzFwY3B5aFdqOHI1MWw4RFR6eVpEVUZWcTlXcC8KOFdkYXZ3OURwS0srckIxcEM5dFJadXF2ZjNQTkJ2UjdXdnZNS3FXTCtGZi9LdXhTMk9yenRCSXBSeVJJQTNwNgowTnBDbVVNenJObDRBRGNDZ2dFQkFQOHg5Y255dlVXbHl4b01xKzVLaTVpS3ppVzBBb1loWlZVajFLN3BRNktPCkwxeEs1UHg0WS9aTmNlNGs0c3F4azBkZmpCT3J1eFdYSk1oMGhYSHhPVzhBUDBSYU1waWc5YVh4MWkwVXp6anIKbEZLZWtoNzNrQWpFcHBYN0FGcVRHOHF2NUJ2azArRWI1cGhPNFM4ZFlSQngxdzdSKzYvVisxS3JuYm1LOTE1RQoxbVBWTWtzQ2dlNnFwUGs4aHhWeHVhcStrOWR3eVVVSXhQR2w1NW5BU0hyN1JYaldYa3pkallIZk8vZHFxSGwwCkc3MWx2QmN5dUMwOERsMW5LVE5KQlAvY0pEeGVNVlVha25NM00yN2FNeitPSmRnTUE2YVV1V3pYS3NIWHdBeHkKeWM0ZWJwZEdKcVkrRGRhNGNLckV6VTQ4T2hTM1l6cGxZcXlyemtqeXBtMENnZ0VCQU5PSFUvbEdHNjlldmlNZwo0VjRQckRqUDdPVmRCVEtFVWFkMWNvZzB0bkxkWkFpTlVITkMrdGt6SmlKbWYzU0dCMDV4 WjMxUDIxOFI0YVZOClRVYlJLc1dmNFlwdERCT0RXQ0RCTnVDR2FoMmFQR0Jvd3R2M0ZEb0Y4MnEzL01MY1Ixc0RJNEFidVBtUFJaVFEKMklSWHhQUTRFTXdZZFg2NHFONjd5VzBUd09uQ1BWRVpVVFFXcEthaWxORkJKMFNQUGNLdkRIaEswWjRPaTY1OAphKzYxUVJ5WDNNdTZHeTUyblVrdmlabXBucjZEbTc1ak9meGdRRzNSQ0hjVVpveklDZWZpOTVXZmplbktHWmZYCjdyY3Zlck1nSVl5bVRpNHVzWlpLU0Z3SDFuVjlEWDg3d2ROOVowUytDN01Yd0J6RkZrN3R1bEVrRkt0VHJNU1gKZDNNbzlaOENnZ0VCQUptaFhCVHRtMkI0aFNMemlmWDB6c0ZRbnZZM3ZtVTlhYUd0NW5ZK1c4ZGF6Y3hFRWtLagp2NW9oMlhyQ25mM2tsWU9jVTlucndyOG10TEF3NWIxSXVZakhuMDdvOWhqSW5kbi9FeThrbmZQb2J1eW1KZFdhCnVBMXZSZEo0dnlmSDlDMDdZcG9nVWlYdEJBK2hQUk4rSGxjbUVaQU1mZmJIWFh2UmNTeW9LbStJQllFb2NoU1MKTitLVXZLaUc5ZFBGR1Q5ZlorV0ZNc1hxbDVYYWlPa1l4d215aDRwTms0dTJ2Vi80SWtPNjVCM3A5bUU1QTlqNwpZY0ZwckVReXZLenhRcDg5eWxyRllmR1lBNTUxRnZPZlRNYndMbnc0RDJLVG1tV1p5MVhUS3Z6VGhnWWViL21GCjgvS1UzUVB0R1hiVTc3d3ZxYythNzVQU1FXc3VLd3ZqZUtrQ2dnRUFWMHlBbG9YQ2hicnQrYml3MTVsek50dmwKUktWbFlyeXNoWlJ4VTNoRU9yTllPY042WkwxazZLWnUySVhWckJBVjlQNlVPbmdaK1VxUVpibHluSGVZcTZteApDWU1HSjVaNVZkYzJEYXVVaUNkc zRybkd5U0pKWGFpUC82WVlxbEJvWkMvdnlHK3dBTy9ybThtdFlrR1hhdEc5CkRqMkJqd2l5Z0ROQUlKZ2l0OHZYTWM0RFlvK3Z5WkJUZVRNb21RNkFITDJiNUc2NE81ZWxHU08yZzd3SnJ6QysKdjhCZUIyTCs3U0x0amNVTEsydktoWFNnVE9EZTUwWWU4UVFBVTRYM1lIQStPc3pGa1pLWVNGaEtnV3hzNDdzUApIRGU1ZS92NjFNRDFnbkdaVjBrS1hIRHhwVWExNkpTKzlya3Z2UEF6aVQwNzltUnJ4YTliUnhmd2lZS3FQUT09Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== kind: Secret metadata: labels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd name: etcd-jwt-token namespace: apisix type: Opaque --- apiVersion: apps/v1 kind: StatefulSet metadata: generation: 1 labels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd name: etcd namespace: apisix spec: podManagementPolicy: Parallel replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd serviceName: etcd-headless template: metadata: annotations: checksum/token-secret: 60156acd10c1cc700c72beb227921dd1c20bf596700362f42d7ab98b32f67a44 prometheus.io/port: "2379" prometheus.io/scrape: "true" labels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd namespaces: - apisix topologyKey: kubernetes.io/hostname weight: 1 containers: - env: - name: BITNAMI_DEBUG value: "false" - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: ETCDCTL_API value: "3" - name: ETCD_ON_K8S value: "yes" - name: ETCD_START_FROM_SNAPSHOT value: "no" - name: ETCD_DISASTER_RECOVERY value: "no" - name: ETCD_NAME value: $(MY_POD_NAME) - name: ETCD_DATA_DIR value: /bitnami/etcd/data - name: ETCD_LOG_LEVEL value: info - name: ALLOW_NONE_AUTHENTICATION value: "yes" - name: ETCD_AUTH_TOKEN value: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m - name: ETCD_ADVERTISE_CLIENT_URLS value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2379 - name: ETCD_LISTEN_CLIENT_URLS value: http://[::]:2379 - name: ETCD_INITIAL_ADVERTISE_PEER_URLS value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380 - name: ETCD_LISTEN_PEER_URLS value: http://[::]:2380 - name: ETCD_INITIAL_CLUSTER_TOKEN value: etcd-cluster-k8s - name: ETCD_INITIAL_CLUSTER_STATE value: new - name: ETCD_INITIAL_CLUSTER value: etcd-0=http://etcd-0.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380 - name: ETCD_CLUSTER_DOMAIN value: etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local - name: MY_STS_NAME value: etcd image: {{ .harbor_k_host }}/{{ .image_etcd }} imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /opt/bitnami/scripts/etcd/prestop.sh livenessProbe: failureThreshold: 5 initialDelaySeconds: 60 periodSeconds: 30 successThreshold: 1 tcpSocket: port: 2379 timeoutSeconds: 5 name: etcd ports: - containerPort: 2379 name: client protocol: TCP - containerPort: 2380 name: peer protocol: TCP resources: requests: cpu: 1 memory: 1G ephemeral-storage: 10Gi limits: cpu: 1 memory: 1G ephemeral-storage: 10Gi securityContext: runAsNonRoot: true runAsUser: 1001 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /bitnami/etcd name: data - mountPath: /opt/bitnami/etcd/certs/token/ name: etcd-jwt-token readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1001 serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 volumes: - name: etcd-jwt-token secret: defaultMode: 256 secretName: etcd-jwt-token updateStrategy: type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: nfs-controller volumeMode: Filesystem status: phase: Pending --- apiVersion: v1 kind: Service metadata: annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" labels: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd name: etcd-headless namespace: apisix spec: clusterIP: None clusterIPs: - None internalTrafficPolicy: Cluster ipFamilies: - IPv6 - IPv4 ipFamilyPolicy: RequireDualStack ports: - name: client port: 2379 protocol: TCP targetPort: client - name: peer port: 2380 protocol: TCP targetPort: peer selector: app.kubernetes.io/instance: etcd app.kubernetes.io/name: etcd sessionAffinity: None type: ClusterIP ``` ### Environment - APISIX version (run `apisix version`): - Operating system (run `uname -a`): - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
