hyt05 commented on issue #7026:
URL: https://github.com/apache/apisix/issues/7026#issuecomment-1255934641
I also had the same problem
apisix daemonset configuration file
```
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: apisix
namespace: default
spec:
selector:
matchLabels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
template:
metadata:
annotations:
checksum/config:
7fcdf2496b815f03e6da46a2b4f9ccf62862af978f875828bb86d12f75c94107
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
spec:
containers:
- image: 192.168.101.30/devops/apache/apisix:2.13.1-alpine
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 30
name: apisix
ports:
- containerPort: 80
hostPort: 80
name: http
protocol: TCP
- containerPort: 443
hostPort: 443
name: tls
protocol: TCP
- containerPort: 9180
hostPort: 9180
name: admin
protocol: TCP
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/local/apisix/conf/config.yaml
name: apisix-config
subPath: config.yaml
- mountPath: /etc/localtime
name: timezone
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
initContainers:
- command:
- sh
- -c
- until nc -z apisix-etcd.default.svc.cluster.local 2379; do echo
waiting
for etcd `date`; sleep 2; done;
image: busybox:1.28
imagePullPolicy: IfNotPresent
name: wait-etcd
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: apisix
serviceAccountName: apisix
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
operator: Exists
- effect: NoSchedule
operator: Exists
volumes:
- configMap:
defaultMode: 420
name: apisix
name: apisix-config
- hostPath:
path: /etc/localtime
type: ""
name: timezone
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
```
ServiceAccount
```
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: apisix
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: apisix
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: apisix
subjects:
- kind: ServiceAccount
name: apisix
namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: apisix
```
configmap
```
apiVersion: v1
data:
config.yaml: |-
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
apisix:
node_listen: 80 # APISIX listening port
enable_heartbeat: true
enable_admin: true
enable_admin_cors: true
enable_debug: false
enable_dev_mode: false # Sets nginx
worker_processes to 1 if set to true
enable_reuseport: true # Enable nginx
SO_REUSEPORT switch if set to true.
enable_ipv6: false # Enable nginx IPv6 resolver
config_center: etcd # etcd: use etcd to store
the config value
# yaml: fetch the config
value from local yaml file `/your_path/conf/apisix.yaml`
#proxy_protocol: # Proxy Protocol configuration
# listen_http_port: 9181 # The port with proxy protocol for
http, it differs from node_listen and port_admin.
# This port can only receive http
request with proxy protocol, but node_listen & port_admin
# can only receive http request. If
you enable proxy protocol, you must use this port to
# receive http request with proxy
protocol
# listen_https_port: 9182 # The port with proxy protocol for
https
# enable_tcp_pp: true # Enable the proxy protocol for tcp
proxy, it works for stream_proxy.tcp option
# enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the
upstream server
proxy_cache: # Proxy Caching configuration
cache_ttl: 10s # The default caching time if the
upstream does not specify the cache time
zones: # The parameters of a cache
- name: disk_cache_one # The name of the cache,
administrator can be specify
# which cache to use by name in the
admin api
memory_size: 50m # The size of shared memory, it's
used to store the cache index
disk_size: 1G # The size of disk, it's used to
store the cache data
disk_path: "/tmp/disk_cache_one" # The path to store the cache data
cache_levels: "1:2" # The hierarchy levels of a cache
# - name: disk_cache_two
# memory_size: 50m
# disk_size: 1G
# disk_path: "/tmp/disk_cache_two"
# cache_levels: "1:2"
allow_admin: #
http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
- 127.0.0.1/24
- 0.0.0.0/0
# - "::/64"
port_admin: 9180
# Default token when use API to call for Admin API.
# *NOTE*: Highly recommended to modify this value to protect APISIX's
Admin API.
# Disabling this configuration item means that the Admin API does not
# require any authentication.
admin_key:
# admin: can everything for configuration data
- name: "admin"
key: edd1c9f034335f136f87ad84b625c8f1
role: admin
# viewer: only can view configuration data
- name: "viewer"
key: 4054f7cf07e344346cd3f287985e76a2
role: viewer
router:
http: 'radixtree_uri' # radixtree_uri: match route by
uri(base on radixtree)
# radixtree_host_uri: match route by
host + uri(base on radixtree)
ssl: 'radixtree_sni' # radixtree_sni: match route by
SNI(base on radixtree)
stream_proxy: # TCP/UDP proxy
only: false
tcp: # TCP proxy port list
- 9100
udp: # UDP proxy port list
- 9200
dns_resolver_valid: 30
resolver_timeout: 5
ssl:
enable: true
enable_http2: true
listen_port: 443
ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
ssl_ciphers:
"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
control:
ip: 127.0.0.1
port: 9091
discovery:
kubernetes: { }
nginx_config: # config for render the template to
genarate nginx.conf
error_log: "/dev/stderr"
error_log_level: "debug" # warn,error
worker_rlimit_nofile: 20480 # the number of files a worker process
can open, should be larger than worker_connections
event:
worker_connections: 10620
http:
enable_access_log: true
access_log: "/dev/stdout"
access_log_format: "$remote_addr - $remote_user [$time_local]
$http_host \"$request\" $status $body_bytes_sent $request_time
\"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status
$upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\""
access_log_format_escape: default
keepalive_timeout: 60s # timeout during which a keep-alive
client connection will stay open on the server side.
client_header_timeout: 60s # timeout for reading client request
header, then 408 (Request Time-out) error is returned to the client
client_body_timeout: 60s # timeout for reading client request
body, then 408 (Request Time-out) error is returned to the client
send_timeout: 10s # timeout for transmitting a response
to the client.then the connection is closed
underscores_in_headers: "on" # default enables the use of
underscores in client request header fields
real_ip_header: "X-Real-IP" #
http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
real_ip_from: #
http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
- 127.0.0.1
- 'unix:'
http_configuration_snippet: |-
server_names_hash_bucket_size 128;
proxy_buffer_size 128k;
proxy_buffers 32 256k;
proxy_busy_buffers_size 256k;
etcd:
host: # it's possible to define
multiple etcd hosts addresses of the same etcd cluster.
- "http://apisix-etcd.default.svc.cluster.local:2379";
prefix: "/apisix" # apisix configurations prefix
timeout: 30 # 30 seconds
plugins: # plugin list
- api-breaker
- authz-keycloak
- basic-auth
- batch-requests
- consumer-restriction
- cors
- echo
- fault-injection
- grpc-transcode
- hmac-auth
- http-logger
- ip-restriction
- ua-restriction
- jwt-auth
- kafka-logger
- key-auth
- limit-conn
- limit-count
- limit-req
- node-status
- openid-connect
- authz-casbin
- prometheus
- proxy-cache
- proxy-mirror
- proxy-rewrite
- redirect
- referer-restriction
- request-id
- request-validation
- response-rewrite
- serverless-post-function
- serverless-pre-function
- sls-logger
- syslog
- tcp-logger
- udp-logger
- uri-blocker
- wolf-rbac
- zipkin
- traffic-split
- gzip
- real-ip
- ext-plugin-pre-req
- ext-plugin-post-req
- server-info
- ldap-auth
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
plugin_attr:
prometheus:
export_uri: /apisix/prometheus/metrics
metric_prefix: apisix_
enable_export_server: true
export_addr:
ip: 127.0.0.1
port: 9092
plugin_attr:
server-info:
report_ttl: 60
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: Helm
name: apisix
namespace: default
```
route
```
{
"uri": "/*",
"name": "asd",
"host": "test.test.cn",
"upstream": {
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "http",
"discovery_type": "kubernetes",
"pass_host": "pass",
"service_name": "default/nginx:80",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
},
"status": 1
}
```
Error log generated for a single request
```
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] route.lua:72:
create_radixtree_uri_router(): insert uri route:
{"id":"426665771179967242","host":"hyt.test.cn","methods":["GET","POST","PUT","DELETE","PATCH","HEAD","OPTIONS","CONNECT","TRACE"],"uri":"\/*","upstream":{"nodes":[{"weight":1,"port":9443,"host":"hub-console.hub-tenant"}],"type":"roundrobin","pass_host":"pass","timeout":{"send":6,"connect":6,"read":6},"keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320},"scheme":"https","parent":{"createdIndex":994,"has_domain":true,"modifiedIndex":994,"key":"\/apisix\/routes\/426665771179967242","value":{"id":"426665771179967242","host":"hyt.test.cn","methods":"table:
0x7f1839175cc8","uri":"\/*","upstream":"table:
0x7f1839141328","priority":0,"status":1,"update_time":1663842217,"name":"minio","create_time":1663842217},"clean_handlers":{},"update_count":0,"orig_modifiedIndex":994},"hash_on":"vars"},"priority":0,"status":1,"update_time":1663842217,"name":"minio","create_time":16
63842217}, client: 192.168.100.88, server: _, request: "GET / HTTP/1.1", host:
"test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] route.lua:72:
create_radixtree_uri_router(): insert uri route:
{"id":"426790842154353418","host":"test.test.cn","uri":"\/*","upstream":{"type":"roundrobin","keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320},"hash_on":"vars","pass_host":"pass","timeout":{"send":6,"connect":6,"read":6},"discovery_type":"kubernetes","scheme":"http","parent":{"createdIndex":1011,"has_domain":false,"modifiedIndex":1108,"key":"\/apisix\/routes\/426790842154353418","value":{"id":"426790842154353418","host":"test.test.cn","uri":"\/*","upstream":"table:
0x7f1833bed410","priority":0,"update_time":1663921234,"status":1,"name":"asd","create_time":1663916765},"clean_handlers":{},"update_count":0,"orig_modifiedIndex":1108},"service_name":"default\/nginx:80"},"priority":0,"update_time":1663921234,"status":1,"name":"asd","create_time":1663916765},
client: 192.168.100.88, server: _, request: "GET / HTTP/1.1", host:
"test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] route.lua:94:
create_radixtree_uri_router(): route items:
[{"paths":"\/*","handler":"function:
0x7f1833b36678","priority":0,"hosts":"hyt.test.cn","methods":["GET","POST","PUT","DELETE","PATCH","HEAD","OPTIONS","CONNECT","TRACE"]},{"handler":"function:
0x7f1833b39a98","priority":0,"paths":"\/*","hosts":"test.test.cn"}], client:
192.168.100.88, server: _, request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:346:
pre_insert_route(): path: / operator: <=, client: 192.168.100.88, server: _,
request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:234:
insert_route(): insert route path: / dataprt: 1, client: 192.168.100.88,
server: _, request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:346:
pre_insert_route(): path: / operator: <=, client: 192.168.100.88, server: _,
request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:564:
match_route_opts(): hosts match: false, client: 192.168.100.88, server: _,
request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:564:
match_route_opts(): hosts match: true, client: 192.168.100.88, server: _,
request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:488:
compare_param(): pcre pat: \/((.|\n)*), client: 192.168.100.88, server: _,
request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] init.lua:388:
http_access_phase(): matched route:
{"createdIndex":1011,"has_domain":false,"modifiedIndex":1108,"key":"\/apisix\/routes\/426790842154353418","value":{"id":"426790842154353418","host":"test.test.cn","uri":"\/*","upstream":{"type":"roundrobin","keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320},"hash_on":"vars","pass_host":"pass","timeout":{"send":6,"connect":6,"read":6},"discovery_type":"kubernetes","scheme":"http","parent":{"createdIndex":1011,"has_domain":false,"modifiedIndex":1108,"key":"\/apisix\/routes\/426790842154353418","value":"table:
0x7f1833bed3c8","clean_handlers":{},"update_count":0,"orig_modifiedIndex":1108},"service_name":"default\/nginx:80"},"priority":0,"update_time":1663921234,"status":1,"name":"asd","create_time":1663916765},"clean_handlers":"table:
0x7f1833bed4b0","update_count":0,"orig_modifiedIndex":1108}, client:
192.168.100.88, server: _, request: "GET / HTTP/1.1", host: "test.test.cn
"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] init.lua:317: nodes(): get
empty endpoint version from discovery DICT default/nginx, client:
192.168.100.88, server: _, request: "GET / HTTP/1.1", host: "test.test.cn"
2022/09/23 16:22:41 [error] 45#45: *111554 [lua] init.lua:512:
http_access_phase(): failed to set upstream: no valid upstream node: nil,
client: 192.168.100.88, server: _, request: "GET / HTTP/1.1", host:
"test.test.cn"
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:564:
match_route_opts(): hosts match: false, client: 192.168.100.88, server: _,
request: "GET /favicon.ico HTTP/1.1", host: "test.test.cn", referrer:
"http://test.test.cn/";
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:564:
match_route_opts(): hosts match: true, client: 192.168.100.88, server: _,
request: "GET /favicon.ico HTTP/1.1", host: "test.test.cn", referrer:
"http://test.test.cn/";
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] radixtree.lua:488:
compare_param(): pcre pat: \/((.|\n)*), client: 192.168.100.88, server: _,
request: "GET /favicon.ico HTTP/1.1", host: "test.test.cn", referrer:
"http://test.test.cn/";
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] init.lua:388:
http_access_phase(): matched route:
{"createdIndex":1011,"has_domain":false,"modifiedIndex":1108,"key":"\/apisix\/routes\/426790842154353418","value":{"id":"426790842154353418","host":"test.test.cn","uri":"\/*","upstream":{"type":"roundrobin","keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320},"hash_on":"vars","pass_host":"pass","timeout":{"send":6,"connect":6,"read":6},"discovery_type":"kubernetes","scheme":"http","parent":{"createdIndex":1011,"has_domain":false,"modifiedIndex":1108,"key":"\/apisix\/routes\/426790842154353418","value":"table:
0x7f1833bed3c8","clean_handlers":{},"update_count":0,"orig_modifiedIndex":1108},"service_name":"default\/nginx:80"},"priority":0,"update_time":1663921234,"status":1,"name":"asd","create_time":1663916765},"clean_handlers":"table:
0x7f1833bed4b0","update_count":0,"orig_modifiedIndex":1108}, client:
192.168.100.88, server: _, request: "GET /favicon.ico HTTP/1.1", host: "t
est.test.cn", referrer: "http://test.test.cn/";
2022/09/23 16:22:41 [info] 45#45: *111554 [lua] init.lua:317: nodes(): get
empty endpoint version from discovery DICT default/nginx, client:
192.168.100.88, server: _, request: "GET /favicon.ico HTTP/1.1", host:
"test.test.cn", referrer: "http://test.test.cn/";
2022/09/23 16:22:41 [error] 45#45: *111554 [lua] init.lua:512:
http_access_phase(): failed to set upstream: no valid upstream node: nil,
client: 192.168.100.88, server: _, request: "GET /favicon.ico HTTP/1.1", host:
"test.test.cn", referrer: "http://test.test.cn/";
```
- APISIX version (run `apisix version`): 2.13.1-alpine
- Operating system (run `uname -a`): 5.15.0-39-generic
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
openresty/1.19.9.1
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): bitnami/etcd:3.4.18-debian-10-r14
- APISIX Dashboard version, if relevant: apisix-dashboard:2.13-alpine
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
