prajith-nair opened a new issue, #7008: URL: https://github.com/apache/apisix/issues/7008
### Current Behavior **apisix_conf/config.yaml** `# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # apisix: node_listen: 9080 # APISIX listening port enable_ipv6: false allow_admin: # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow - 0.0.0.0/0 # We need to restrict ip access rules for security. 0.0.0.0/0 is for test. admin_key: - name: "admin" key: xxxxx role: admin # admin: manage all configuration data # viewer: only can view configuration data - name: "viewer" key: xxxxx role: viewer enable_control: true control: ip: "0.0.0.0" port: 9092 etcd: host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster. - "http://labindia01_etcd:2379"; # multiple etcd address, if your etcd cluster enables TLS, please use https scheme, # e.g. https://127.0.0.1:2379. prefix: /apisix # apisix configurations prefix timeout: 30 # 30 seconds #resync_delay: 5 # when sync failed and a rest is needed, resync after the configured seconds plus 50% random jitter #health_check_timeout: 10 # etcd retry the unhealthy nodes after the configured seconds health_check_retry: 2 # etcd retry time that only affects the health check, default 2 #user: root # root username for etcd #password: 5tHkHhYkjr6cQY # root password for etcd tls: # To enable etcd client certificate you need to build APISIX-OpenResty, see # https://apisix.apache.org/docs/apisix/how-to-build/#step-6-build-openresty-for-apache-apisix #cert: /path/to/cert # path of certificate used by the etcd client #key: /path/to/key # path of key used by the etcd client verify: false # whether to verify the etcd endpoint certificate when setup a TLS connection to etcd, # the default value is true, e.g. the certificate will be verified strictly. #sni: # the SNI for etcd TLS requests. If missed, the host part of the URL will be used. plugin_attr: prometheus: export_addr: ip: "0.0.0.0" port: 9091` etcd.conf.yml : `# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This is the configuration file for the etcd server. # Human-readable name for this member. name: 'default' # Path to the data directory. data-dir: # Path to the dedicated wal directory. wal-dir: # Number of committed transactions to trigger a snapshot to disk. snapshot-count: 10000 # Time (in milliseconds) of a heartbeat interval. heartbeat-interval: 100 # Time (in milliseconds) for an election to timeout. election-timeout: 1000 # Raise alarms when backend size exceeds the given quota. 0 means use the # default quota. quota-backend-bytes: 0 # List of comma separated URLs to listen on for peer traffic. listen-peer-urls: http://localhost:2380 # List of comma separated URLs to listen on for client traffic. listen-client-urls: http://localhost:2379 # Maximum number of snapshot files to retain (0 is unlimited). max-snapshots: 5 # Maximum number of wal files to retain (0 is unlimited). max-wals: 5 # Comma-separated white list of origins for CORS (cross-origin resource sharing). cors: # List of this member's peer URLs to advertise to the rest of the cluster. # The URLs needed to be a comma-separated list. initial-advertise-peer-urls: http://localhost:2380 # List of this member's client URLs to advertise to the public. # The URLs needed to be a comma-separated list. advertise-client-urls: http://localhost:2379 # Discovery URL used to bootstrap the cluster. discovery: # Valid values include 'exit', 'proxy' discovery-fallback: 'proxy' # HTTP proxy to use for traffic to discovery service. discovery-proxy: # DNS domain used to bootstrap initial cluster. discovery-srv: # Initial cluster configuration for bootstrapping. initial-cluster: # Initial cluster token for the etcd cluster during bootstrap. initial-cluster-token: 'etcd-cluster' # Initial cluster state ('new' or 'existing'). initial-cluster-state: 'new' # Reject reconfiguration requests that would cause quorum loss. strict-reconfig-check: false # Accept etcd V2 client requests enable-v2: true # Enable runtime profiling data via HTTP server enable-pprof: true # Valid values include 'on', 'readonly', 'off' proxy: 'off' # Time (in milliseconds) an endpoint will be held in a failed state. proxy-failure-wait: 5000 # Time (in milliseconds) of the endpoints refresh interval. proxy-refresh-interval: 30000 # Time (in milliseconds) for a dial to timeout. proxy-dial-timeout: 1000 # Time (in milliseconds) for a write to timeout. proxy-write-timeout: 5000 # Time (in milliseconds) for a read to timeout. proxy-read-timeout: 0 client-transport-security: # Path to the client server TLS cert file. cert-file: # Path to the client server TLS key file. key-file: # Enable client cert authentication. client-cert-auth: false # Path to the client server TLS trusted CA cert file. trusted-ca-file: # Client TLS using generated certificates auto-tls: false peer-transport-security: # Path to the peer server TLS cert file. cert-file: # Path to the peer server TLS key file. key-file: # Enable peer client cert authentication. client-cert-auth: false # Path to the peer server TLS trusted CA cert file. trusted-ca-file: # Peer TLS using generated certificates. auto-tls: false # Enable debug-level logging for etcd. debug: false logger: zap # Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd. log-outputs: [stderr] # Force to create a new one member cluster. force-new-cluster: false auto-compaction-mode: periodic auto-compaction-retention: "1"` **docker-compose.yml** ` apisix: hostname: "{{.Service.Name}}" image: apache/apisix:2.13.1-alpine restart: always depends_on: - etcd #network_mode: host ports: - "9080:9080/tcp" - "9091:9091/tcp" - "9443:9443/tcp" - "9092:9092/tcp" configs: - source: apisix-config target: /usr/local/apisix/conf/config.yaml mode: 0444 networks: my-stack-network: aliases: - apisix deploy: restart_policy: condition: any update_config: # ensure at least one instance runs when updating order: start-first resources: limits: memory: 100M reservations: memory: 100M placement: constraints: - ${SERVICE_CONSTRAINT:-node.role==worker} logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:${RSYSLOG_EXPOSED_PORT:-10514}" tag: "apisix/{{.Name}}/container.log" syslog-facility: local2 syslog-format: rfc3164 etcd: hostname: "{{.Service.Name}}" image: bitnami/etcd:3.5.4 restart: always volumes: - etcd_data:/bitnami/etcd environment: ETCD_ENABLE_V2: "true" ALLOW_NONE_AUTHENTICATION: "yes" ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2379"; ETCD_LISTEN_CLIENT_URLS: "http://labindia01_etcd:2379"; ports: - "2379:2379/tcp" networks: my-stack-network: aliases: - etcd deploy: restart_policy: condition: any update_config: # ensure at least one instance runs when updating order: start-first resources: limits: memory: 100M reservations: memory: 100M placement: constraints: - ${SERVICE_CONSTRAINT:-node.role==worker} logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:${RSYSLOG_EXPOSED_PORT:-10514}" tag: "etcd/{{.Name}}/container.log" syslog-facility: local2 syslog-format: rfc3164` I can ping @labindia01_etcd:/opt/bitnami/etcd/bin$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters **10.0.32.31 labindia01_etcd** From Other container on same network, I can ping labindia01_etcd hostname /usr/local/apisix-dashboard # ping labindia01_etcd PING labindia01_etcd (10.0.32.21): 56 data bytes 64 bytes from 10.0.32.21: seq=0 ttl=64 time=0.106 ms 64 bytes from 10.0.32.21: seq=1 ttl=64 time=0.086 ms 64 bytes from 10.0.32.21: seq=2 ttl=64 time=0.075 ms 64 bytes from 10.0.32.21: seq=3 ttl=64 time=0.105 ms 64 bytes from 10.0.32.21: seq=4 ttl=64 time=0.072 ms However, defining hostname in config.yaml of apisix like below, would lead to apisix service to fail with ``` etcd: host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster. - "http://labindia01_etcd:2379"; # multiple etcd address, if your etcd cluster enables TLS, please use https scheme, # e.g. https://127.0.0.1:2379. prefix: /apisix # apisix configurations prefix timeout: 30 # 30 seconds tls: verify: false ``` replacing etcd container IP 10.0.32.31 in api config.yml works as expected. ### Expected Behavior apisix service should come up with hostname defined in config.yml !@labindia01_etcd:/opt/bitnami/etcd/bin$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters **10.0.32.31 labindia01_etcd** **Before** etcd: host: - "http://127.0.0.1:2379"; ETCD_HOST being hostname of container (labindia01_etcd) should resolve in apisix **After** etcd: host: - "http://${{ETCD_HOST}}:2379"; ### Error Logs ``` labindia01_apisix.1.p3m2ok88e93c@xxxxxx | labindia01_apisix.1.u2pazna86jp8@xxxxxx | /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd labindia01_apisix.1.p3m2ok88e93c@xxxxxx | /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd labindia01_apisix.1.u2pazna86jp8@xxxxxx | 2022/05/09 05:36:53 [error] 1#1: init_by_lua error: /usr/local/apisix/apisix/core/config_etcd.lua:816: attempt to concatenate local 'err' (a nil value) labindia01_apisix.1.p3m2ok88e93c@xxxxxx | 2022/05/09 05:36:39 [error] 1#1: init_by_lua error: /usr/local/apisix/apisix/core/config_etcd.lua:816: attempt to concatenate local 'err' (a nil value) labindia01_apisix.1.u2pazna86jp8@xxxxxx | stack traceback: labindia01_apisix.1.p3m2ok88e93c@xxxxxx | stack traceback: labindia01_apisix.1.u2pazna86jp8@xxxxxx | /usr/local/apisix/apisix/core/config_etcd.lua:816: in function 'init' labindia01_apisix.1.u2pazna86jp8@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.p3m2ok88e93c@xxxxxx | /usr/local/apisix/apisix/core/config_etcd.lua:816: in function 'init' labindia01_apisix.1.u2pazna86jp8@xxxxxx | init_by_lua:9: in main chunk labindia01_apisix.1.p3m2ok88e93c@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.p3m2ok88e93c@xxxxxx | init_by_lua:9: in main chunk labindia01_apisix.1.u2pazna86jp8@xxxxxx | nginx: [error] init_by_lua error: /usr/local/apisix/apisix/core/config_etcd.lua:816: attempt to concatenate local 'err' (a nil value) labindia01_apisix.1.u2pazna86jp8@xxxxxx | stack traceback: labindia01_apisix.1.u2pazna86jp8@xxxxxx | /usr/local/apisix/apisix/core/config_etcd.lua:816: in function 'init' labindia01_apisix.1.p3m2ok88e93c@xxxxxx | nginx: [error] init_by_lua error: /usr/local/apisix/apisix/core/config_etcd.lua:816: attempt to concatenate local 'err' (a nil value) labindia01_apisix.1.tto2u9kbjiqm@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.p3m2ok88e93c@xxxxxx | stack traceback: labindia01_apisix.1.tto2u9kbjiqm@xxxxxx | init_by_lua:9: in main chunk labindia01_apisix.1.p3m2ok88e93c@xxxxxx | /usr/local/apisix/apisix/core/config_etcd.lua:816: in function 'init' labindia01_apisix.1.hy71wbvj83bk@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.hy71wbvj83bk@xxxxxx | init_by_lua:9: in main chunk labindia01_apisix.1.p3m2ok88e93c@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.p3m2ok88e93c@xxxxxx | init_by_lua:9: in main chunk labindia01_apisix.1.u2pazna86jp8@xxxxxx | /usr/local/apisix/apisix/init.lua:84: in function 'http_init' labindia01_apisix.1.u2pazna86jp8@xxxxxx | init_by_lua:9: in main chunk ``` ### Steps to Reproduce 1. Deploy stack using below docker-compose.yml ` version: "3.8" services: ################################################################################ # APISIX for traffic management apisix-dashboard: hostname: "{{.Service.Name}}" image: apache/apisix-dashboard:2.10.1-alpine restart: always ports: - "9000:9000" configs: - source: apisixdashboard-config target: /usr/local/apisix-dashboard/conf/conf.yaml mode: 0444 networks: my-stack-network: deploy: restart_policy: condition: any update_config: # ensure at least one instance runs when updating order: start-first resources: limits: memory: 100M reservations: memory: 100M placement: constraints: - ${SERVICE_CONSTRAINT:-node.role==worker} logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:${RSYSLOG_EXPOSED_PORT:-10000}" tag: "apisix-dashboard/{{.Name}}/container.log" syslog-facility: local2 syslog-format: rfc3164 apisix: hostname: "{{.Service.Name}}" image: apache/apisix:2.13.1-alpine #command: /bin/sh -c 'sleep 300' restart: always depends_on: - etcd #network_mode: host ports: - "9080:9080/tcp" - "9091:9091/tcp" - "9443:9443/tcp" - "9092:9092/tcp" configs: - source: apisix-config target: /usr/local/apisix/conf/config.yaml mode: 0444 networks: my-stack-network: aliases: - apisix deploy: restart_policy: condition: any update_config: # ensure at least one instance runs when updating order: start-first resources: limits: memory: 100M reservations: memory: 100M placement: constraints: - ${SERVICE_CONSTRAINT:-node.role==worker} logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:${RSYSLOG_EXPOSED_PORT:-10000}" tag: "apisix/{{.Name}}/container.log" syslog-facility: local2 syslog-format: rfc3164 etcd: hostname: "{{.Service.Name}}" image: bitnami/etcd:3.5.4 restart: always volumes: - etcd_data:/bitnami/etcd environment: ETCD_ENABLE_V2: "true" ALLOW_NONE_AUTHENTICATION: "yes" ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2379"; ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"; ports: - "2379:2379/tcp" networks: my-stack-network: aliases: - etcd deploy: restart_policy: condition: any update_config: # ensure at least one instance runs when updating order: start-first resources: limits: memory: 100M reservations: memory: 100M placement: constraints: - ${SERVICE_CONSTRAINT:-node.role==worker} logging: driver: "syslog" options: syslog-address: "udp://127.0.0.1:${RSYSLOG_EXPOSED_PORT:-10000}" tag: "etcd/{{.Name}}/container.log" syslog-facility: local2 syslog-format: rfc3164 networks: my-stack-network: volumes: etcd_data: driver: local configs: apisix-config: name: ${DOCKER_STACK}_apisix-config-v1 file: ${DEPLOYMENT_HOME}/resources/apisix/apisix_conf/config.yaml apisixdashboard-config: name: ${DOCKER_STACK}_apisixdashboard-config-v1 file: ${DEPLOYMENT_HOME}/resources/apisix/dashboard_conf/conf.yaml ` 2. List the docker service and notice docker service for apisix doesnt comeup as expected. `qoij49qbfiad labindia01_apisix replicated 0/1 apache/apisix:2.13.1-alpine *:9080->9080/tcp, *:9091-9092->9091-9092/tcp, *:9443->9443/tcp` 3. Check docker service log labindia01_apisix for potential errors ### Environment - APISIX version (run `apisix version`): apisix:2.13.1 - Operating system (run `uname -a`): Darwin prajith 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:47:26 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T8101 arm64 - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): nginx version: openresty/1.19.9.1 built by clang 13.1.6 (clang-1316.0.21.2.3) built with OpenSSL 1.1.1n 15 Mar 2022 TLS SNI support enabled - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): etcdctl version: 3.5.4 API version: 3.5 @labindia01_etcd:/opt/bitnami/etcd/bin$ ./etcdctl member list 224bff4639e481b9, started, default, http://172.18.0.6:2380, http://0.0.0.0:2379, false - APISIX Dashboard version, if relevant: 2.10.1-alpine - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): 3.8.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
